Distributional properties of political dogwhistle representations in Swedish BERT

Niclas Hertzberg, Robin Cooper, Elina Lindgren, Björn Rönnerstrand, Gregor Rettenegger, Ellen Breitholtz, Asad SayeedNAACL 2022

TL;DR

This paper explores whether modern language models, specifically a Swedish BERT variant, can capture the subtle, dual-layered meanings of "dogwhistles" in political discourse. By analyzing how people rephrase dogwhistles in a survey, they demonstrate that these nuanced expressions are indeed represented in a way that can be computationally separated, suggesting that NLP models can aid in the detection and analysis of such communication.

What Problem Does This Paper Address?

Political communication often employs "dogwhistles" – phrases that carry a benign message for the general public but a coded, often less acceptable, message for a specific "in-group." This manipulative tactic can deceive voters and undermine democratic processes. While linguistics has studied dogwhistles, automated detection and analysis using NLP techniques are lagging, especially for languages like Swedish with fewer readily available resources. This paper aims to bridge this gap by investigating if a pre-trained Swedish BERT model can represent and distinguish the different interpretations of dogwhistles, thereby paving the way for more scalable analysis.

Methods

The researchers leverage a pre-trained Swedish BERT model, specifically a Swedish sentence-BERT variant, to generate dense vector representations (embeddings) for sentences. They then apply two main approaches:

  1. Unsupervised Clustering: K-means clustering is used to group the sentence embeddings. This is done on both the original 768-dimensional vectors and dimensionality-reduced versions (3D using PCA) to see if distinct clusters emerge for "in-group" vs. "out-group" interpretations of dogwhistles.
  2. Supervised Classification: A linear-kernel Support Vector Machine (SVM) is trained to classify the sentence embeddings into the two categories. This tests the linear separability of the representations.

The effectiveness of these methods is evaluated using cluster purity metrics (Davies-Bouldin and Calinski-Harabasz scores) and SVM performance metrics (F1 scores).

Datasets

The study utilizes a custom dataset created from a survey of 1000 Swedish individuals. Participants were asked to replace suspected dogwhistle phrases in five sentences with what they believed the phrase "really" meant. These responses were then manually annotated to identify whether they captured the "in-group" dogwhistle meaning or a more general interpretation. The paper also references the use of a multi-genre corpus for training the Swedish BERT model.

Key Insights and Ideas

  • Semantic Separation: The study demonstrates that sentence representations from a Swedish BERT model can capture the semantic distinction between "in-group" and "out-group" interpretations of dogwhistles. This suggests that the model has learned to differentiate these subtle meanings.
  • Correlation with Human Judgment: There's a notable correlation between the ease with which human annotators could identify dogwhistles (measured by inter-annotator agreement, IAA) and the separability of the corresponding sentence embeddings in the vector space. Lower IAA generally corresponds to poorer linear separability.
  • SVM as a Detector: A linear SVM can effectively separate the "in-group" and "out-group" responses, indicating that the semantic space learned by BERT allows for a clear distinction, even with limited training data.
  • Clustering vs. Separability: While SVM can linearly separate the groups, K-means clustering doesn't always align perfectly with human IAA scores. This suggests that while the groups might be separable, their internal structure or spread within the semantic space might not always form distinct, easily separable clusters for K-means.

Key Results

  • The SVM generally achieved high F1 scores (e.g., 0.98/0.98 for "enrich" and 0.98/0.97 for "suburban gang") in separating the two types of responses, indicating good linear separability.
  • Dogwhistles with lower inter-annotator agreement (IAA) scores, like "remigration" (0.74/0.55), showed lower F1 scores for both 3-dim (0.72) and 768-dim (0.85) representations, reinforcing the link between human interpretability and model separability.
  • Conversely, dogwhistles with high IAA, such as "suburban gang" (1/1), exhibited very high F1 scores (0.98/0.97).
  • Cluster evaluation metrics (Davies-Bouldin and Calinski-Harabasz) for K-means did not consistently align with IAA scores, suggesting that linear separability (via SVM) might be a more direct indicator of dogwhistle interpretability in this context.

Limitations and Future Work

  • Corpus Influence: The authors acknowledge that the semantic space learned by BERT is heavily influenced by its training corpus. Different corpora could lead to different representations and thus different clustering outcomes.
  • K-means Limitations: K-means struggled to consistently differentiate between in-group and out-group replacements, suggesting that future work could explore alternative clustering methods or ways to model the semantic space to better align with human judgments.
  • Future Directions: The paper suggests investigating how models trained on different text sources might impact the semantic space and dogwhistle representation. They also propose exploring methods to train models that create semantic spaces where algorithms like K-means can retrieve labels directly from the data itself, without explicit supervision.

Summary updated

BibTeX
@inproceedings{hertzberg2022distributional,
  author = {Niclas Hertzberg and Robin Cooper and Elina Lindgren and Björn Rönnerstrand and Gregor Rettenegger and Ellen Breitholtz and Asad Sayeed},
  title = {{Distributional properties of political dogwhistle representations in Swedish BERT}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure