TL;DR
This paper explores whether computational methods for tracking word meaning change over time can predict the use of "dogwhistles" – coded language with hidden meanings. By combining techniques for analyzing lexical semantic change with survey data on how people interpret these phrases, the researchers found that shifts in a word's "in-group" versus "out-group" meaning are indeed strong predictors of its semantic change, particularly when using advanced language models.
What Problem Does This Paper Address?
Political discourse increasingly relies on "dogwhistles," which are phrases that convey one meaning to the general public (out-group) and a different, often more controversial, meaning to a specific subset of the audience (in-group). Identifying and understanding these hidden meanings is crucial for combating hate speech, misinformation, and ensuring democratic legitimacy. While computational methods have been used to analyze text for harmful content, automatically detecting dogwhistles and understanding their semantic evolution remains a challenge. This paper tackles this by investigating whether the temporal dynamics of word meaning change, as captured by distributional semantics, can be leveraged to predict the development and usage of dogwhistles.
Methods
The core of this research lies in combining two main approaches:
- Lexical Semantic Change (LSC) Detection: The authors use established computational methods to track how the meaning of words changes over time. They employ four different pipelines for this:
- SGNS (Skip-gram with Negative Sampling): A classic word embedding model.
- BERT (Bidirectional Encoder Representations from Transformer): A powerful contextualized word embedding model.
- SBERT (Sentence-BERT): A variant of BERT fine-tuned for sentence similarity, which the authors hypothesize might be particularly good at capturing nuanced meanings.
-
mT5 (multilingual Text-to-Text Transfer Transformer): A large, multilingual transformer model. They measure semantic change by comparing word vectors at different time points.
-
In-group/Out-group Meaning Modeling: To capture the dual nature of dogwhistles, the researchers use data from a survey where Swedish residents replaced potential dogwhistle phrases. This allowed them to quantify the "in-group" (hidden) and "out-group" (public) meanings associated with these phrases. They then calculate an "In-group/Out-group Ratio" (IOR) for each phrase at different times.
The central idea is to use linear regression to see if changes in the IOR (ΔIOR) can predict the rate of semantic change (ΔLSC) of dogwhistle expressions.
Datasets
- Flashback Corpus: A large dataset of discussions from a Swedish online forum, used for training the distributional models. The corpus spans from 2000 to 2022 and focuses on political discussions.
- Swedish Replacement Survey: A custom dataset collected by the authors. Participants were asked to replace potential dogwhistle words in sentences, providing data to quantify the in-group and out-group meanings of these phrases.
Key Insights and Ideas
- Dogwhistles are Semantic Chameleons: The study strongly suggests that dogwhistles evolve their meaning over time, specifically by shifting their in-group and out-group dimensions. This dynamic nature is precisely what LSC methods aim to capture.
- Contextual Embeddings are Key: Advanced language models that capture word meaning based on context (like BERT and SBERT) are significantly better at predicting semantic change related to dogwhistles than older, static embedding methods (SGNS).
- SBERT Shines: The SBERT model, specifically designed for sentence similarity, showed the strongest correlation between IOR and semantic change, suggesting that sentence-level representations are particularly effective for understanding dogwhistle nuances.
- The "Why" Matters: The research moves beyond simply detecting semantic change to explaining why it occurs, linking it directly to the strategic use of coded language.
Key Results
- Predictive Power of IOR: For most of the tested models, shifts in the In-group/Out-group Ratio (ΔIOR) were a significant predictor of semantic change (ΔLSC). This indicates that the way a word's in-group and out-group meanings diverge or converge directly influences its overall semantic evolution.
- LLMs Outperform SGNS: The language model-based pipelines (BERT, SBERT, mT5) explained substantially more variability in semantic change compared to the SGNS models.
- SBERT's Superiority: The SBERT pipeline achieved the highest R² values, indicating it best explained the semantic change of dogwhistles.
- Transformer Models Generalize: The multilingual mT5 model performed comparably to the Swedish-specific BERT model, suggesting the effectiveness of large transformer architectures across languages for this task.
- SGNS Nuances: For SGNS models, vector dimensionality and window size played a role, with higher dimensions and a window size of 5 showing the strongest effects.
Limitations and Future Work
- Context Specificity: The study is limited to the political media context of Sweden and a specific set of dogwhistles. Generalizing these findings to other languages, political contexts, or a broader range of dogwhistles requires further testing.
- Data Collection Cost: The survey method used to gather in-group/out-group meanings is effective but costly and difficult to scale.
- Future Directions:
- Scaling Up: Applying the methodology to a larger set of dogwhistles and across different national contexts.
- Alternative Data Sources: Exploring the use of existing online databases of dogwhistle definitions to represent in-group and out-group embeddings, potentially reducing the reliance on costly surveys.
- Broader Applications: Investigating how these methods can be used to detect and combat harmful communication more broadly.
Summary updated