Abstract
In recent years, an increasing number of studies has been published in the newly emerging text-as-data field. More and more scholars in the areas of political and social science are taking advantage of the ever-increasing amount of text available—not only on the internet but also transcriptions of parliamentary debates, newspaper texts, or party manifestos—to address a heterogeneous set of research questions. While this trend has already brought many promising results, it is not free of risks and challenges. In their seminal paper, Grimmer and Stewart (2013) not only discuss the potential of text-as-data approaches but also highlight the pitfalls that arise when applying NLP methods for the investigation of questions from political and social science.
Introduction
This editorial introduces a special issue of the Journal for Language Technology and Computational Linguistics (JLCL), which serves as a synthesis of contributions from the 1st Workshop on Computational Linguistics for the Political and Social Sciences (CPSS 2021). As the "text-as-data" movement gains momentum, it becomes increasingly clear that the intersection of Natural Language Processing (NLP) and social science is not merely a matter of applying existing tools to new datasets. Rather, it requires a deep, interdisciplinary collaboration to address the unique methodological and theoretical challenges inherent in political discourse.
The Challenge of Text-as-Data
While the explosion of digital text—ranging from parliamentary records and party manifestos to social media commentary—offers a goldmine for researchers, applying NLP methods is rarely a "plug-and-play" solution. A fundamental gap exists between the two disciplines: computer scientists often prioritize algorithmic performance and predictive power, whereas social scientists require interpretability, transparency, and theoretical grounding.
Standard unsupervised methods, such as Latent Dirichlet Allocation (LDA) topic modeling, are frequently used but can be "noisy" and difficult to validate against established social science theories. Furthermore, political datasets are often small or highly specialized, leading to significant issues with model robustness. To advance the field, researchers must move beyond using NLP as a mere tool and instead foster a methodology where domain-specific expertise informs model design.
Methodological Innovations
The research presented in this special issue highlights four distinct approaches to bridging this gap:
- Methodological Sensitivity Analysis: One study employs a replication approach to investigate how sensitive machine learning models are to pre-processing steps (e.g., lemmatization, stop-word removal) and data splits. This is particularly critical when working with the "small data" common in political science, where minor technical choices can lead to vastly different substantive conclusions.
- Hybrid Frame Detection: Researchers compared standard unsupervised LDA topic modeling against a hybrid method. By combining static word embeddings with expert-curated framing schemas, the hybrid approach allows for "media framing" analysis—identifying how a story is told—in a way that is more accurate and interpretable than fully automated methods.
- Clustering and Manual Annotation for Social Media: To identify "proto-slogans" on social media, a pipeline was developed to process over 700,000 Facebook comments. By first clustering the data and then applying manual annotation, researchers could detect stylometric patterns and see how supporters adopt the language of political leaders.
- NER as an Alternative to Topic Modeling: There is a growing argument for using Named Entity Recognition (NER) and Named Entity Linking (NEL) as more transparent alternatives to topic models. These tools provide robust insights into which political actors are mentioned and in what context, avoiding the "black box" nature of traditional unsupervised clustering.
Data and Applications
The papers discussed utilize a variety of specialized datasets that reflect the breadth of the field:
- The UNSC-NE Corpus: An extension of the United Nations Security Council debates corpus specifically designed for Named Entity Recognition.
- European Refugee Crisis Archive: A corpus of German newspaper articles from 2014–2018 used to study media framing.
- Populist Social Media Data: A collection of 700,000+ comments from the Facebook pages of Italian leaders Matteo Salvini and Luigi Di Maio.
- General Political Texts: Ongoing work utilizing parliamentary debates and party manifestos to track long-term political trends.
Key Insights and Future Directions
Several critical themes emerge from these contributions. First, the "Small Data" Trap remains a major hurdle; data sparsity in political research means that results are often highly sensitive to training/test splits, necessitating more rigorous validation protocols. Second, Expertise Trumps Pure Automation. The success of hybrid models suggests that "human-in-the-loop" designs are essential for ensuring that NLP outputs align with social science requirements.
Finally, the move toward Interpretability and Transparency is paramount. Whether through the use of NER to track actors or the identification of stylometric "fingerprints" in populist rhetoric, the goal is to create models that social scientists can trust and explain. Future work must continue to focus on model robustness, ensuring that the insights gained from text-as-data are not merely artifacts of specific pre-processing choices but are true reflections of the underlying social phenomena.
Summary updated