How We Do Things With Words: Analyzing Text as Social and Cultural Data

Dong Nguyen, Maria Liakata, Simon DeDeo, Jacob Eisenstein, David Mimno, Rebekah Tromble, Jane Winters

TL;DR

This paper provides a methodological roadmap for interdisciplinary computational text analysis, bridging the gap between computer science, the humanities, and the social sciences. It outlines best practices and challenges across the entire research lifecycle—from formulating insight-driven questions to data curation, operationalization, and validation. The authors emphasize that social and cultural concepts lack an absolute "ground truth," requiring researchers to balance quantitative scale with qualitative nuance.

What Problem Does This Paper Address?

Computational text analysis offers unprecedented scale to study pressing social and cultural issues, such as online hate speech, political polarization, and historical literary shifts. However, traditional computer science approaches typically treat text analysis as a performance-driven optimization problem with a clear, objective "ground truth" (e.g., spam filtering or part-of-speech tagging).

In contrast, the humanities and social sciences deal with "thick," highly contested, and inherently ambiguous concepts where absolute ground truth does not exist. When computer scientists apply rigid models to these fluid concepts, they risk drawing shallow conclusions or latching onto spurious data patterns. This paper addresses the challenge of how to conduct rigorous, insight-driven text analysis that respects the complexity of social and cultural data while leveraging the power of machine learning.

Methods

As a position and review paper, it does not introduce a single new algorithm. Instead, it synthesizes a methodological framework across the text-as-data research lifecycle:

  • Insight-Driven Formulation: Shifting the research focus from automating human tasks to describing and explaining social phenomena.
  • Conceptualization & Operationalization: Distinguishing between a "background concept" (the broad, diverse theoretical meaning of a term) and a "systematized concept" (the specific formulation adopted for the study), which is then translated into measurable "scores."
  • Modeling Trade-offs: Evaluating the strengths and limitations of three primary approaches:
  • Dictionary-based methods: Highly transparent but brittle and domain-dependent.
  • Supervised models: Scalable but highly prone to learning stereotypical or spurious shortcuts from unbalanced data.
  • Unsupervised topic modeling: Excellent for open-ended exploration, acting as a "microscope" where the number of topics is a user-controlled magnification level rather than an inherent property of the text.
  • The Validation Loop: Advocating for an iterative validation process that pairs machine outputs with qualitative "close reading" to continually refine codebooks, annotations, and algorithms.

Datasets

The paper does not introduce new datasets or benchmarks. Instead, it critiques the data sources typically utilized in this domain, categorizing them into two main types:

  • Born-Digital Data: Social media data (e.g., Twitter, Reddit) that offer massive scale and unobtrusive measurement. The authors highlight critical issues with these datasets, including commercial API biases, lack of informed consent, and demographic skews (e.g., Reddit's user base leaning toward young white men).
  • Digitized Historical Archives: Analogue sources converted via Optical Character Recognition (OCR) (e.g., Google Books, the Gutenberg archive, historical newspapers). These suffer from uneven digitization choices, OCR errors, and hidden composition biases (such as the over-representation of scientific articles in Google Books).

Key Insights and Ideas

  • Ambiguity is a Feature, Not a Bug: In the humanities and social sciences, ambiguity and differing interpretations are prized. Disagreements between human annotators are not just "noise" to be averaged out; they often signal conceptual weaknesses or meaningful real-world complexities.
  • The Danger of Spurious Features: Supervised models easily latch onto dataset-specific noise. For example, the authors note a cited study where the strongest predictive feature for identifying sexism on Twitter was the name of an Australian TV show, simply because users frequently posted sexist remarks about its contestants.
  • No Absolute Ground Truth: Human labels are heavily influenced by ideological priors, priming, and context. Treating human annotations as an flawless "gold standard" can severely compromise model validity.
  • Errors as Opportunities: Model "failures" or unexpected omissions (the "dog that didn't bark") serve as excellent starting points for closer qualitative analysis, helping researchers discover entirely new patterns they didn't know to look for.

Key Results

Because this is a conceptual framework paper, it does not present new quantitative benchmarks. Instead, it highlights qualitative success stories from cited literature to demonstrate the power of its framework:

  • It showcases how a matching design (treatment vs. control groups) allowed researchers to rigorously demonstrate that banning specific hate-speech subreddits led to an overall decline in hate speech across the rest of Reddit.
  • It highlights how combining quantitative text modeling with close reading allowed literary scholars to discover a previously unrecognized genre—the "conversional novel"—within a corpus of 450 18th-century novels.

Limitations and Future Work

The authors identify several open challenges for the field:

  • API Gatekeeping: The increasing restriction of public APIs by major platforms (like Facebook and Twitter) threatens the accessibility and reproducibility of born-digital research.
  • Defining "Balance": There is currently no objective definition of what constitutes a "balanced" dataset in social media research, making it difficult to mitigate temporal and demographic biases.
  • The Black Box Problem: Modern deep learning approaches (e.g., word embeddings, neural networks) reduce the need for manual feature engineering but severely damage interpretability. The authors emphasize that future work must focus on developing transparent, post-hoc explanation methods so that insight-driven analysis remains viable.

Summary updated

Abstract

In this article we describe our experiences with computational text analysis. We hope to achieve three primary goals. First, we aim to shed light on thorny issues not always at the forefront of discussions about computational text analysis methods. Second, we hope to provide a set of best practices for working with thick social and cultural concepts. Our guidance is based on our own experiences and is therefore inherently imperfect. Still, given our diversity of disciplinary backgrounds and research practices, we hope to capture a range of ideas and identify commonalities that will resonate for many. And this leads to our final goal: to help promote interdisciplinary collaborations. Interdisciplinary insights and partnerships are essential for realizing the full potential of any computational text analysis that involves social and cultural concepts, and the more we are able to bridge these divides, the more fruitful we believe our work will be.
BibTeX
@article{nguyen2020how,
  author = {Dong Nguyen and Maria Liakata and Simon DeDeo and Jacob Eisenstein and David Mimno and Rebekah Tromble and Jane Winters},
  title = {{How We Do Things With Words: Analyzing Text as Social and Cultural Data}},
  journal = {Frontiers Artif. Intell.},
  year = {2020},
  doi = {10.3389/frai.2020.00062},
}
Figure