Data Education & Best Practices

The Carpentries (Software Carpentry, Data Carpentry, Library Carpentry) offer a suite of self-guided lessons in foundational coding and data science skills, perfect for researchers and scholars. The materials are designed to be accessible to absolute beginners, covering essential tools like R, Python, and SQL in a hands-on, easy-to-follow format. Each lesson takes approximately 3-4 hours to complete with the R and Python programming languages taking between 7-8 hours.

• Tidy Data
  • Good data organization is the foundation of any research project. Typically, we organize data in spreadsheets in a way that we as humans want to work with the data. However, computers require data to be organized in particular ways. In order to use tools that make computation more efficient, such as programming languages like R or Python, we need to structure our data the way that computers need the data. Since this is where most research projects start, this is a great place to start too.    
• The Unix Shell
  • The Unix Shell is a tool that allows users to perform complex and powerful tasks, often with just a few keystrokes or lines of code. It helps users automate repetitive tasks and easily combine smaller tasks into larger, more powerful workflows. Use of the shell is fundamental to a wide range of advanced computing tasks, including high-performance computing. This lesson will introduce you to this powerful tool.    
• Version Control with Git
  • Version control is the lab notebook of the digital world: it’s what professionals use to keep track of what they’ve done and to collaborate with other people. Every large software development project relies on it, and most programmers use it for their small jobs as well. And it isn’t just for software: books, papers, small data sets, and anything that changes over time or needs to be shared can and should be stored in a version control system. Keeping a record of what was changed, when, and why is extremely useful for all researchers if they ever need to come back to the project later on (e.g., a year later, when memory has faded).    
• Python
  • Python is a popular programming language for tasks such as data collection, cleaning, and analysis. Python can help you to create reproducible workflows to accomplish repetitive tasks more efficiently. Software Carpentry has two different Python lessons and Data Carpentry has a lesson for extracting information from image data with Python. Programming with Python teaches you to think like a programmer using Python, while Plotting and Programming with Python teaches you to do data analysis reproducibly using the tools common in the scientific Python community. Both are excellent starting points for a novice, but you might want to choose one that best fits your specific needs and interests.
    • Programming with Python
      • This lesson is a classic, foundational Software Carpentry lesson. It is designed for absolute beginners to programming. Its curriculum is structured around a scientific data analysis task: analyzing patient inflammation data. 
    • Plotting and Programming in Python
      • This lesson is a more modern alternative that is often used in both Software Carpentry and Data Carpentry workshops. It is also for complete beginners, but it introduces more powerful, domain-specific tools earlier in the process.
    • Image Processing with Python
      • This workshop uses Python and a variety of example images to teach the foundational concepts of image processing, and the skills needed to programmatically extract information from image data.
• R
  • R is commonly used in many scientific disciplines for statistical analysis and its array of third-party packages. The emphasis of these lessons is to give attendees a strong foundation in the fundamentals of R, and to teach best practices for scientific computing: breaking down analyses into modular units, task automation, and encapsulation. Software Carpentry also has two different R lessons to choose from.
  • R for Reproducible Scientific Analysis
    • This lesson is the more introductory of the two. It is designed for researchers and others who have no previous programming experience and want to learn how to use R for data analysis.
  • Programming with R
    • This lesson is a slightly more advanced offering that targets learners who already have some programming experience, either in R or another language. It goes beyond the basics to delve deeper into programming concepts and best practices.
• SQL
  • Structured Query Language, or SQL (sometimes pronounced “sequel”), is a powerful language used to interrogate and manipulate relational databases. It is not a general programming language that you can use to write an entire program. However, SQL queries can be called from programming languages to let any program interact with databases. There are several variants of SQL, but all support the same basic statements, and SQLite and DB Browser are used in this lesson. DB Browser for SQLite provides a graphical user interface for connecting to and interacting with a SQLite database.
• OpenRefine
  • OpenRefine is a powerful free, open source tool that uses your web browser as a graphical interface. It is a power tool for working with messy data: cleaning it; transforming it from one format into another; and extending it with web services and external data. OpenRefine empowers everyone to meaningfully engage with data by providing an accessible open source tool and nurturing a diverse, supportive community.

There are even more Carpentries lessons to explore in the Data Carpentry and Library Carpentry curriculums. Data Carpentry Lessons are discipline focused (Ecology, Genomics, Geospatial, Social Sciences), while Library Carpentry’s lessons apply concepts of software development and data science to information science and the humanities.

Here are two great options for learning Geographic Information Systems (GIS) tools. You can get started with industry-standard tools like ESRI (ArcGIS Pro), which offers powerful features and official tutorials, or dive into the free and flexible QGIS, which is an open-source alternative with a large community of support.

• ESRI Training Courses for ArcGIS
  • Esri E-Learning courses are online, self-paced, step by step lessons that cover a variety of topics related to GIS applications and technology. Currently, there are 500+ course selections to choose from that are included with our standard subscription. UB affiliates are eligible to enroll free of charge. Users may complete the lessons at their own pace and courses are valid for one year. As a general rule, please allow 2-4 hours to complete workshops and approximately 18-24 hours to complete courses
• QGIS Training Manual
  • QGIS is a free, open-source GIS tool that works on most operating systems. The tool allows users to run spatial analyses with geospatial data. The QGIS Training Manual includes documentation and lessons for using QGIS that include sample datasets to test out each process.

The Programming Historian provides peer-reviewed, step-by-step tutorials specifically tailored for humanists and social scientists. From digitizing historical maps to analyzing textual data, their lessons equip you with the technical know-how to tackle new research questions. Whether you're aiming to automate a tedious task or explore new forms of digital analysis, these platforms provide the perfect starting point for your self-guided learning journey. Here is a sample of the more than 100 lessons at The Programming Historian.

• Preserving Your Research Data
  • This lesson will suggest ways in which researchers can document and structure their research data so as to ensure it remains useful in the future.
• Understanding Regular Expressions
  • This lesson uses advanced find-and-replace to apply computational thinking to a historical document structured as a prose table. The goal is to expose researchers to pattern matching, a skill useful for word processors and a foundation for learning programming languages.
• Introduction to Jupyter Notebooks
  • Jupyter notebooks provide an environment where you can freely combine human-readable narrative with computer-readable code. This lesson describes how to install the Jupyter Notebook software, how to run and create Jupyter notebook files, and contexts where Jupyter notebooks can be particularly helpful.
• Intro to Google Maps and Google Earth
  • Google My Maps is an intuitive tool for plotting historical data on a map, allowing you to add points, lines, and polygons or import data from a spreadsheet. Your maps are saved to your Google account and can be shared or exported for use in Google Earth or QGIS. This is the first in a series of five GIS lessons that also includes learning QGIS.
• Course Analysis with Voyant tools
  • This lesson is designed as a beginner-friendly introduction to corpus analysis. You will learn the basic steps of creating a corpus along with learning the main metrics of quantitative text analysis. Voyant, a web-based platform that does not require installation and works in any browser will be used to do this.
• Omeka
  • Open-source web publishing platform for sharing digital collections and creating media-rich online exhibits. The Programming Historian has three lessons for using Omeka.
    • Installing Omeka
      • This lesson will teach you how to install your own copy of Omeka.
    • Up and Running with Omeka.net
      • Omeka.net makes it easy to create websites that show off collections of items.
    • Creating an Omeka Exhibit
      • Now that you’ve added items to your Omeka site and grouped them into collections, you’re ready for the next step: taking your users on a guided tour through the items you’ve collected.
• Jekyll
  • Jekyll is an open-source static site generator that transforms plain text, specifically Markdown and HTML files, into a complete, ready-to-serve static website or blog. GitHub Pages is powered by Jekyll, so you can easily deploy your site using GitHub for free—custom domain name and all. The Programming Historian has two lessons for learning and using Jekyll.
    • Building a static website with Jekyll and GitHub Pages
      • This lesson will help you create entirely free, easy-to-maintain, preservation-friendly, secure website over which you have full control, such as a scholarly blog, project website, or online portfolio.
    • Running a Collaborative Research Website and Blog with Jekyll and GitHub
      • In this lesson you will be introduced to the challenges and opportunities that Jekyll, a popular, static site generator, offers for publishing collaborative, ongoing research online.
• Computer Vision for the Humanities: An Introduction to Deep Learning for Image Classification in Two Parts
  • The first lesson introduces deep learning based computer vision methods for humanities research. Using a dataset of historical newspaper advertisements and the fastai Python library, the lesson walks through the pipeline of training a computer vision model to perform image classification.
  • The second lesson digs deeper into the details of training a deep learning based computer vision model. It covers some challenges one may face due to the training data used and the importance of choosing an appropriate metric for your model. It presents some methods for evaluating the performance of a model.
• R
  • There are 11 R lessons at the Programming Historian. Below is a sample of what you will find.
    • Data Wrangling and Management in R
      • This tutorial explores how scholars can organize ‘tidy’ data, understand R packages to manipulate data, and conduct basic data analysis.
    • Basic Text Processing With R
      • Learn how to use R to analyze high-level patterns in texts, apply stylometric methods over time and across authors, and use summary methods to describe items in a corpus.
    • Text Mining YouTube Comment Data with Wordfish in R
      • In this lesson, you will learn how to download YouTube video comments and use the R programming language to analyze the dataset with Wordfish, an algorithm designed to identify opposing ideological perspectives within a corpus.
    • Introduction to MySQL with R
      • This lesson will help you store large amounts of historical data in a structured manner, search and filter that data, and visualize some of the data as a graph.
    • Making an Interactive Web Application with R and Shiny
      • This lesson demonstrates how to build an interactive webmap using R and the Shiny library. In the lesson, you will design and implement a simple application, consisting of a slider which allows a user to select a date range, and display a set of corresponding points, on an interactive map.
• Python
  • There are 36 Python lessons at the Programming Historian. If you are new to Python, you can start off with Python Introduction and Installation, which is the first lesson in a 15 lesson Python series.
    • Python Introduction and Installation
      • In this opening module you will install the Python programming language, the Beautiful Soup HTML/XML parser, and a text editor. Once everything is installed, you will get your toes wet with some simple programming that provides immediate results.
• NLTK
  • The Natural Language Toolkit (NLTK) is a popular Python module dedicated to natural language processing. The Programming Historian has 7 lessons where you will learn and use NLTK. Below are a couple lessons to get you started.
    • Introduction to stylometry with Python
      • In this lesson you will learn to conduct ‘stylometric analysis’ on texts and determine authorship of disputed texts. The lesson covers three methods: Mendenhall’s Characteristic Curves of Composition, Kilgariff’s Chi-Squared Method, and John Burrows’ Delta Method.
    • Sentiment Analysis for Exploratory Data Analysis
      • In this lesson you will learn to conduct ‘sentiment analysis’ on texts and to interpret the results. This is a form of exploratory data analysis based on natural language processing. You will learn to install all appropriate software and to build a reusable program that can be applied to your own texts.