Measuring and Modeling Language Change

Jacob EisensteinNAACL 2019 · OpenReview

Abstract

This paper synthesizes computational techniques and social science methodologies to address fundamental questions regarding linguistic evolution. By leveraging time-stamped document corpora, we explore how language serves as a proxy for social reality, tracking phenomena such as political polarization, semantic shifts, and the lifecycle of lexical innovations. We bridge the gap between computational linguistics and the quantitative social sciences by introducing a framework that includes dynamic word embeddings, Hawkes processes, and causal inference. This tutorial-based approach provides researchers with the tools to quantify social trends and linguistic leadership through the lens of temporal text data.

Introduction

Language is a moving target. Words shift in meaning, political groups drift apart in their rhetoric, and new terms emerge while others vanish. While researchers in the social sciences and digital humanities have long sought to use these changes to understand societal shifts, a technical disconnect often persists. Computational linguists possess powerful tools for processing text but may lack the statistical frameworks—such as causal inference—common in the social sciences. Conversely, social scientists may be unfamiliar with modern NLP techniques like dynamic word embeddings.

This paper addresses this gap by synthesizing recent computational techniques into a coherent framework for corpus-based temporal analysis. We move beyond simple word frequency counts to explore how sophisticated modeling can extract meaningful social and linguistic insights from a timeline of text.

Methodological Framework

To effectively measure language change, we must employ a multi-disciplinary toolkit that accounts for the nuances of temporal data.

1. Dynamic Word Embeddings

Unlike static embeddings, dynamic word embeddings are parameterized by time. This allows researchers to track "lexical semantics" within a vector space, identifying the specific moments when a word’s meaning begins to drift. For example, diachronic embeddings can pinpoint when "money laundering" transitioned into a common legal concept or when the term "gay" shifted from its historical meaning of "happy" to its contemporary usage.

2. Modeling Influence and Innovation

To understand the dynamics of community interaction, we utilize Hawkes Processes and Granger Causation. These methods model "leader-follower" dynamics by analyzing the timing of specific word usage in debates or scholarly discourse. By mathematically estimating who sets the terms of a discussion, we can identify discursive influence.

Furthermore, Survival Analysis—a method borrowed from medical statistics—is applied to lexical innovations (neologisms). This allows us to predict which new words will "stick" within a community and which are merely ephemeral trends based on early adoption patterns.

3. Causal Inference

To move beyond mere correlation, we employ Interrupted Time Series frameworks. This is critical for evaluating the impact of specific events or interventions. For instance, researchers can determine whether a specific platform policy caused a decrease in hate speech or if the trend was already occurring independently of the intervention.

Data and Applications

The efficacy of these methods is demonstrated across a diverse array of datasets, ranging from social media to historical archives:

  • Political Polarization: By analyzing US Legislative floor speeches and primary debate transcripts, we can measure the "textual distance" between political parties. Research confirms that American politicians are increasingly polarized, a trend clearly reflected in their diverging vocabularies.
  • Social Media Dynamics: Twitter sentiment data reveals diurnal (time-of-day) and seasonal mood variations across cultures, while Reddit data provides a laboratory for studying the lifecycle of hate speech and the effectiveness of moderation policies.
  • Historical Trends: The Google Books corpus and historical newspaper archives allow for the longitudinal study of cultural shifts over decades or centuries.

Key Insights: Language as a Social Sensor

A central premise of this work is that language change is not merely an internal linguistic process; it is a proxy for social reality. Tracking the evolution of text allows us to quantify phenomena that traditional metrics, such as voting records, might miss. Whether measuring the effectiveness of a subreddit ban or identifying linguistic leaders in a political movement, the integration of temporal modeling with NLP provides a high-resolution view of social change.

Limitations and Future Directions

While this framework provides a robust starting point, certain boundaries remain. This approach focuses on corpus-based analysis and excludes simulation-based models or "apparent time" studies (which compare age groups at a single point in time). Additionally, modeling "bad language" on the internet and refining the measurement of discursive influence remain active areas of research. Future work must continue to integrate these computational tools with the nuanced theoretical models of historical linguistics to better understand the mechanisms driving our evolving discourse.

Conclusion

By bridging the gap between NLP and the social sciences, researchers can transform vast archives of time-stamped text into actionable insights. The techniques outlined here—from dynamic embeddings to causal inference—provide a roadmap for understanding not just how language changes, but what those changes reveal about the world around us.

Summary updated

Abstract

This tutorial is designed to help researchers answer the following sorts of questions: - Are people happier on the weekend? - What was 1861’s word of the year? - Are Democrats and Republicans more different than ever? - When did “gay” stop meaning “happy”? - Are gender stereotypes getting weaker, stronger, or just different? - Who is a linguistic leader? - How can we get internet users to be more polite and objective? Such questions are fundamental to the social sciences and humanities, and scholars in these disciplines are increasingly turning to computational techniques for answers. Meanwhile, the ACL community is increasingly engaged with data that varies across time, and with the social insights that can be offered by analyzing temporal patterns and trends. The purpose of this tutorial is to facilitate this convergence in two main ways: 1. By synthesizing recent computational techniques for handling and modeling temporal data, such as dynamic word embeddings, the tutorial will provide a starting point for future computational research. It will also identify useful tools for social scientists and digital humanities scholars. 2. The tutorial will provide an overview of techniques and datasets from the quantitative social sciences and the digital humanities, which are not well-known in the computational linguistics community. These techniques include vector autoregressive models, multiple comparisons corrections for hypothesis testing, and causal inference. Datasets include historical newspaper archives and corpora of contemporary political speech.
BibTeX
@inproceedings{eisenstein2019measuring,
  author = {Jacob Eisenstein},
  title = {{Measuring and Modeling Language Change}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=B1Wf17Wd-r},
}
Figure