Introduction
While large contextualized language models like BERT dominate most modern NLP tasks, recent research reveals a surprising limitation: they struggle to detect subtle "connotative" shifts. For instance, the word "climate" might imply a "hoax" in one political community but a "threat" in another. This paper demonstrates that traditional static embeddings and a novel "Masked Target Prediction" approach—which analyzes model predictions rather than internal vector geometry—are significantly more effective at capturing these nuanced social differences.
The Challenge of Subtle Semantic Shifts
Lexical Semantic Change Detection (LSCD) typically focuses on "extreme" diachronic shifts over long periods, such as the evolution of "apple" from a fruit to a technology brand. However, social scientists are increasingly interested in subtle synchronic shifts: how different communities use the same words with divergent associations or "vibes" at the same point in time.
These shifts are often "associative" (linked to different topics) or "evaluative" (positive vs. negative sentiment) rather than complete changes in dictionary definition. While contextualized models like BERT are often assumed to be superior due to their sensitivity to context, this study investigates their efficacy in detecting these politically charged nuances.
Experimental Setup and Datasets
The study utilizes Reddit data from 2016–2022 across two primary linguistic contexts:
- Dutch Political Discourse: Comparing Poldersocialisme (left-wing) and Forum_Democratie (right-wing).
- English Political Discourse: Comparing hillaryclinton and The_Donald.
The researchers selected 12 target words per language, categorized into "associative" terms (e.g., immigration, climate) and "evaluative" terms (e.g., socialist, liberal).
Methodology
The authors compared three distinct methodological families:
- Static Representations: Traditional models (PPMI and Word2Vec/SGNS) that create a single vector per word per community.
- Contextualized Representations: Models like BERT and XLM-R using Average Pairwise Distance (APD) to measure vector shifts across datasets.
- Masked Target Prediction (Novel): This behavioral approach masks the target word and asks the model to predict the top 10 substitutes. The frequency distributions of these substitutes are then compared using Jensen-Shannon Divergence (JSD).
Key Results
The experimental results highlight a stark performance gap between traditional/behavioral methods and standard contextualized vector approaches.
- Subtle Shift Detection: Static models and the new Masked Target Prediction method achieved 100% accuracy (24/24 correct binary decisions) in distinguishing between same-community and different-community word usage.
- Contextualized Failure: Standard BERT-based vector methods (APD) performed poorly, often yielding results barely better than random guessing for subtle shifts.
- Interpretability: Masked Target Prediction offers high transparency. For the word "income," the model predicted "money" and "property" in The_Donald, but "social" and "gender" in hillaryclinton, providing immediate qualitative insight into community worldviews.
Discussion: Why BERT Struggles
The core insight is that BERT’s internal representations are "too strong." The information encoded during its massive pretraining phase acts as a dominant signal; the local context of a specific subreddit is often insufficient to significantly alter the word's internal vector. Even in synthetic tests where "guns" was manually replaced with "taxes," vector-based models failed to detect the shift.
By contrast, Masked Target Prediction bypasses these "frozen" vectors by looking at what the model expects to see in a given context, successfully revealing community-specific associations that geometric distance obscures.
Conclusion
This research suggests that for social science applications involving subtle semantic nuances, "older" static models or behavioral prediction-based methods are currently more reliable than standard contextualized vector distances. Future work will explore refining the Masked Target Prediction method by adjusting the number of predicted substitutes and incorporating prediction probabilities.
Summary updated