digital-humanities

Files (2)

• SKILL.md
• dh-tools-guide.md · references/

SKILL.md (first lines)

# Digital Humanities Methods and Tools

## Overview

Digital humanities (DH) applies computational methods to the study of human culture, history, language, and society. It is not the replacement of humanistic inquiry with algorithms but the augmentation of interpretive scholarship with tools that can reveal patterns invisible to unaided reading, connect dispersed archives, visualize historical processes, and make cultural heritage accessible to broader audiences.

This skill covers the major computational methods used in humanities research: text mining and natural language processing (topic modeling with LDA and BERTopic, sentiment analysis, named entity recognition), corpus linguistics (concordance, collocation, frequency analysis, keyness), digital archiving and metadata standards (Dublin Core, TEI XML), geographic information systems (GIS) for historical research, network analysis of historical figures and literary characters, stylometry and computational authorship attribution, optical character recognition (OCR) workflows for digitizing historical texts, digital scholarly editions, data visualization for humanities data, distant reading as theorized by Franco Moretti, cultural analytics as developed by Lev Manovich, and the Python ecosystem for humanities computing (spaCy, NLTK, Voyant Tools, AntConc).

The skill is designed for humanities scholars who want to integrate computational methods into their research -- whether they are analyzing Victorian novels, mapping colonial trade networks, studying the evolution of political rhetoric, or building digital archives of endangered languages. No prior programming experience is assumed, though some methods require basic Python or R skills. For each method, the skill describes the intellectual rationale, practical implementation, available tools (from no-code to full programming), and critical perspectives on the method's limitations.

## When to Use This Skill

Use this skill when you need to:

- Apply topic modeling (LDA, BERTopic) to a large text corpus to discover thematic patterns
- Perform sentiment analysis on historical texts, literary works, or political discourse
- Extract named entities (people, places, organizations, dates) from unstructured text
- Conduct corpus linguistics analysis: concordance, collocation, frequency, keyness
- Create or work with digital archives using Dublin Core or TEI XML metadata standards
- Use GIS to map historical events, trade routes, migration patterns, or spatial narratives
- Build and analyze networks of historical figures, literary characters, or intellectual influence
- Attribute authorship of disputed texts using stylometry and computational methods
- Digitize historical documents using OCR (Tesseract, Kraken, Transkribus)
- Create digital scholarly editions with critical apparatus
- Visualize humanities data using Palladio, Gephi, or custom tools
- Apply distant reading methods to analyze literary trends across large corpora
- Conduct cultural analytics on visual media, social media, or digital culture
- Work with Python NLP tools (spaCy, NLTK) for humanities text analysis

## Core Capabilities

### Text Mining and NLP for Humanities

Natural language processing (NLP) provides computational tools for analyzing text at scales impossible for human readers. In humanities research, NLP is not a replacement for close reading but a complement that can identify patterns across thousands or millions of texts, guide the selection of passages for close analysis, and test hypotheses about language change, genre conventions, and cultural trends.

#### Topic Modeling

Topic modeling uses unsupervised machine learning to discover latent thematic structure in document collections. The two dominant approaches are Latent Dirichlet Allocation (LDA) and BERTopic.

**Latent Dirichlet Allocation (LDA):**

LDA (Blei, Pritchard, & Ng, 2003) models each document as a mixture of topics, and each topic as a distribution over words. It is a bag-of-words model -- word order does not matter.

**LDA workflow:**

1. **Corpus preparation** -- Collect and clean texts (remove headers, footers, metadata)
2. **Preprocessing** -- Tokenize, lowercase, remove stop words, lemmatize
3. **Feature extraction** -- Create document-term matrix or bag-of-words representation
4. **Model training** -- Run LDA with specified number of topics (k)
5. **Evaluation** -- Assess coherence scores, inspect topic-word distributions
6. **Interpretation** -- Label topics based on high-probability words and representative documents
7. **Analysis** -- Track topic proportions across time, genres, authors, or other categories

**Example: LDA topic from a corpus of 19th-century British novels**

```
Topic 7 (labeled "Domestic Life"):
  Top words: room, house, door, table, fire, chair, window, sat,
             morning, evening, bed, garden, dinner, tea, kitchen

  Top documents: Cranford (Gaskell), Middlemarch (Eliot),
                 North and South (Gaskell)

  Interpretation: This topic captures domestic settings and daily
  routines. Its prevalence increases in novels by women authors
  and in novels published after 1850, suggesting a shift toward
  domestic realism in mid-Victorian fiction.
```

**Choosing the number of topics (k):**
- Coherence scores (higher is better) -- compute for k = 5, 10, 15, 20, 25, 30, 40, 50
- Human interpretability -- can you label each topic meaningfully?
- Research question alignment -- does the granularity match your analytical needs?
- Common range: 15-50 topics for corpora of 1,000-10,000 documents

**BERTopic:**

BERTopic (Grootendorst, 2022) uses transformer-based sentence embeddings (BERT) to create document representations, then clusters them using HDBSCAN and extracts topic representations using c-TF-IDF. Unlike LDA, it captures semantic meaning beyond individual words.

**BERTopic advantages over LDA:**
- Captures semantic similarity (not just word co-occurrence)
- Handles short texts better (tweets, abstracts, metadata)
- Produces more coherent topics on modern text
- Does not require specifying the number of topics in advance

**BERTopic Python implementation:**

```python
from bertopic import BERTopic
from sentence_transformers import SentenceTransformer

# Use a sentence transformer model
embedding_model = SentenceTransformer("all-MiniLM-L6-v2")

# Initialize and fit BERTopic
topic_model = BERTopic(
    embedding_model=embedding_model,
    min_topic_size=10,
    nr_topics="auto"
)

topics, probs = topic_model.fit_transform(documents)

# Inspect topics
topic_model.get_topic_info()

# Visualize topic distribution
topic_model.visualize_topics()

# Track topics over time
topics_over_time = topic_model.topics_over_time(
    documents, timestamps
)
topic_model.visualize_topics_over_time(topics_over_time)
```

#### Sentiment Analysis

Sentiment analysis classifies text by emotional valence (positive, negative, neutral) or more specific emotional categories. In humanities research, it is used to study emotional arcs in novels, shifts in political rhetoric, audience reception in reviews, and emotional expression across historical periods.

From LUNARTECH-X/superpowers, MIT.

More from LUNARTECH-X/superpowers