Theory-Grounded Computational Text Analysis

Arya D. McCarthy, Giovanna Maria Dora DoreACL 2023 · DOI

Introduction

Computational text analysis currently faces a "descriptive" bottleneck. While researchers are increasingly adept at reporting what patterns exist within large-scale datasets, there is a growing deficit in explaining why these patterns occur. This position paper argues that for the field of computational social science to mature, it must pivot from mere pattern matching toward "integrative" research grounded in established social science theory.

The Problem: Methods Over Meaning

The "honeymoon period" of computational social science has seen a fruitful exchange between Natural Language Processing (NLP) and social science. However, a significant gap has emerged. Much of the current research is purely descriptive, utilizing complex models to catalog evidence—such as identifying common topics in social media discourse—without addressing the broader "So what?" question.

By prioritizing "state-of-the-art" performance and objective reductionism, computer science often overlooks the contextual and critical frameworks necessary for interpreting human behavior. Without theory, the field risks measuring the rate of a falling apple without ever conceptualizing gravity.

Methodology and Taxonomy

To evaluate the state of the field, the authors conducted a meta-analysis of approximately 60 papers published in major NLP venues (e.g., ACL, EMNLP, NAACL). They categorized these works into two primary buckets:

  1. Descriptive: Findings that uncover patterns (e.g., word clusters or censorship rates) but fail to link them to broader social or communication theories.
  2. Integrative: Research that offers causal explanations or tests specific theory-based hypotheses, enabling predictive insights.

The authors trace this "empirical turn" back to 1996, noting a historical shift in NLP from linguistic theory toward machine learning performance.

Analysis of the Field

The paper’s qualitative critique highlights a trend where engineering-focused goals often overshadow scientific inquiry.

Descriptive Research

Many high-profile papers provide rigorous data analysis but miss opportunities to ground their findings in theoretical frameworks. For instance, studies on polarization or media manipulation often omit explicit links to "propaganda theory" or "social conflict theory."

Examples of descriptive research that lacks explicit theoretical grounding.

Integrative Research

Conversely, "gold standard" works demonstrate the power of theory-grounded analysis. A notable example is Nelson (2021), which utilized word embeddings to confirm specific features of identity theory within 19th-century narratives.

Examples of integrative research that successfully bridges theory and method.

Key Insights: The "Two Masters" Problem

Interdisciplinary researchers often face the "Two Masters" dilemma: computer science reviewers demand methodological complexity, while social science reviewers require theoretical depth. This tension frequently results in "safe," descriptive papers that fail to satisfy either discipline fully. Theory should not be viewed as a constraint but as an empowering organizing principle that moves the field beyond "mathiness" toward a genuine understanding of social processes.

Conclusion and Future Directions

The authors urge the community to rebalance the pendulum by:

  • Prioritizing research questions that are theory-affirming or theory-building.
  • Embracing a wider variety of methods, including qualitative and ethnographic approaches, when they better suit the research question.
  • Encouraging major NLP venues to value interdisciplinary depth over incremental methodological improvements.

By integrating social science theory into computational workflows, the field can move beyond describing the "fall" and start identifying the "gravity" of social phenomena.

Summary updated

BibTeX
@inproceedings{mccarthy2023theory,
  author = {Arya D. McCarthy and Giovanna Maria Dora Dore},
  title = {{Theory-Grounded Computational Text Analysis}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.acl-short.136},
}
Figure