Methodological Insights in Detecting Subtle Semantic Shifts with Contextualized and Static Language Models

Sanne Hoeken, Özge Alaçam, Antske Fokkens, Pia SommerauerEMNLP 2023 · DOI

Introduction

While large contextualized language models like BERT dominate most modern NLP tasks, recent research reveals a surprising limitation: they struggle to detect subtle "connotative" shifts. For instance, the word "climate" might imply a "hoax" in one political community but a "threat" in another. This paper demonstrates that traditional static embeddings and a novel "Masked Target Prediction" approach—which analyzes model predictions rather than internal vector geometry—are significantly more effective at capturing these nuanced social differences.

The Challenge of Subtle Semantic Shifts

Lexical Semantic Change Detection (LSCD) typically focuses on "extreme" diachronic shifts over long periods, such as the evolution of "apple" from a fruit to a technology brand. However, social scientists are increasingly interested in subtle synchronic shifts: how different communities use the same words with divergent associations or "vibes" at the same point in time.

These shifts are often "associative" (linked to different topics) or "evaluative" (positive vs. negative sentiment) rather than complete changes in dictionary definition. While contextualized models like BERT are often assumed to be superior due to their sensitivity to context, this study investigates their efficacy in detecting these politically charged nuances.

Experimental Setup and Datasets

The study utilizes Reddit data from 2016–2022 across two primary linguistic contexts:

  • Dutch Political Discourse: Comparing Poldersocialisme (left-wing) and Forum_Democratie (right-wing).
  • English Political Discourse: Comparing hillaryclinton and The_Donald.

The researchers selected 12 target words per language, categorized into "associative" terms (e.g., immigration, climate) and "evaluative" terms (e.g., socialist, liberal).

Target words and corpus sizes for Dutch and English political discourse datasets.

Methodology

The authors compared three distinct methodological families:

  1. Static Representations: Traditional models (PPMI and Word2Vec/SGNS) that create a single vector per word per community.
  2. Contextualized Representations: Models like BERT and XLM-R using Average Pairwise Distance (APD) to measure vector shifts across datasets.
  3. Masked Target Prediction (Novel): This behavioral approach masks the target word and asks the model to predict the top 10 substitutes. The frequency distributions of these substitutes are then compared using Jensen-Shannon Divergence (JSD).

Key Results

The experimental results highlight a stark performance gap between traditional/behavioral methods and standard contextualized vector approaches.

Performance comparison across static, contextualized, and masked target prediction methods.
  • Subtle Shift Detection: Static models and the new Masked Target Prediction method achieved 100% accuracy (24/24 correct binary decisions) in distinguishing between same-community and different-community word usage.
  • Contextualized Failure: Standard BERT-based vector methods (APD) performed poorly, often yielding results barely better than random guessing for subtle shifts.
  • Interpretability: Masked Target Prediction offers high transparency. For the word "income," the model predicted "money" and "property" in The_Donald, but "social" and "gender" in hillaryclinton, providing immediate qualitative insight into community worldviews.

Discussion: Why BERT Struggles

The core insight is that BERT’s internal representations are "too strong." The information encoded during its massive pretraining phase acts as a dominant signal; the local context of a specific subreddit is often insufficient to significantly alter the word's internal vector. Even in synthetic tests where "guns" was manually replaced with "taxes," vector-based models failed to detect the shift.

By contrast, Masked Target Prediction bypasses these "frozen" vectors by looking at what the model expects to see in a given context, successfully revealing community-specific associations that geometric distance obscures.

Conclusion

This research suggests that for social science applications involving subtle semantic nuances, "older" static models or behavioral prediction-based methods are currently more reliable than standard contextualized vector distances. Future work will explore refining the Masked Target Prediction method by adjusting the number of predicted substitutes and incorporating prediction probabilities.

Summary updated

BibTeX
@inproceedings{hoeken2023methodological,
  author = {Sanne Hoeken and Özge Alaçam and Antske Fokkens and Pia Sommerauer},
  title = {{Methodological Insights in Detecting Subtle Semantic Shifts with Contextualized and Static Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.237},
}
Figure