TL;DR
This paper introduces SemScale, a novel text scaling algorithm that leverages semantic word embeddings to better capture political positions from text, outperforming traditional frequency-based methods. By incorporating word meaning, SemScale offers more robust and interpretable political analysis, especially for complex datasets.
What Problem Does This Paper Address?
Political scientists often use "text scaling" to infer the latent political positions of actors (like parties or politicians) directly from their written or spoken words. Traditional methods, such as Wordfish and Wordscores, primarily rely on word frequencies (the "bag-of-words" approach). While effective, these methods have a significant limitation: they treat words as discrete symbols and struggle when similar political ideas are expressed using different vocabulary, or when similar words are used to express different ideas. This can lead to inaccurate position estimations. The authors aim to overcome this by developing a text scaling method that understands the meaning of words, not just their frequency.
Methods
The core of the paper is the introduction of SemScale, a semantically-aware text scaling algorithm. SemScale has two main components:
- Text Representation Module: This module converts texts into numerical representations. The default approach uses averaged word embeddings. Word embeddings are dense, multi-dimensional vectors that capture word meaning, where words with similar meanings have similar vectors. The paper also explores using traditional TF-IDF vectors and party-specific embeddings (Party2Vec) for comparison.
- Graph-Based Clustering Module: This module uses the text representations to build a similarity graph where texts are nodes and edge weights represent their semantic similarity (calculated using cosine similarity). SemScale then employs a Harmonic Function Label Propagation (HFLP) algorithm to propagate position scores across this graph. Two "pivot texts" (the least similar pair) are assigned extreme positions (-1 and 1), and these scores are then diffused to all other texts.
The authors also explore variations, including using different linguistic features (like only nouns or verbs), different text representations, and comparing HFLP with PageRank for graph clustering. They also introduce a weakly supervised version of SemScale (SemScores).
Datasets
The primary dataset used is a collection of speeches from the European Parliament (EP). This dataset was extended from previous work to include speeches from five languages (English, German, French, Italian, Spanish) and two legislative terms (5th and 6th). The authors also used party manifestos from the Comparative Manifestos Project for additional validation.
A new benchmarking dataset for text scaling was created from the EP speeches, which is made publicly available.
Key Insights and Ideas
- Semantics Matter: The most significant insight is that incorporating word meaning (via embeddings) leads to demonstrably better political text scaling than relying solely on word frequencies. This directly addresses a core limitation of existing methods.
- Flexibility of SemScale: The modular design of SemScale is a key advantage. It allows for easy swapping of text representation methods and graph clustering algorithms, facilitating experimentation and adaptation.
- Linguistic Features Can Help (But Carefully): Filtering input to specific linguistic features (like nouns or verbs) can sometimes improve scaling results, suggesting that not all words contribute equally to capturing political dimensions. However, the impact is nuanced and depends on the specific features and algorithm.
- Temporal Drift is a Challenge: SemScale, being sensitive to semantic shifts, may struggle when texts span long temporal periods, potentially mistaking topical similarities for positional ones.
Key Results
- Superior Performance: SemScale consistently outperformed the baseline Wordfish algorithm in capturing political positions related to European integration, showing significantly higher correlation scores with expert-derived ground truth across multiple languages and legislative terms. For example, in Table 2, SemScale achieved an average Pearson correlation of 0.32 (5th Leg) and 0.29 (6th Leg) compared to Wordfish's 0.15 and 0.16, respectively.
- Robustness: SemScale demonstrated good robustness across different languages.
- Manifesto Scaling: While SemScale generally performed well on party manifestos, its performance varied more across countries and election settings compared to Wordfish, especially in multi-election scenarios.
- Weak Supervision Boost: A weakly supervised version of SemScale (SemScores) significantly improved scaling quality when provided with just two reference texts at the extremes of the scale, outperforming even the supervised Wordscores algorithm in some cases.
Limitations and Future Work
- Temporal Sensitivity: The authors acknowledge that SemScale's sensitivity to semantic shifts makes it less suitable for texts spanning long time periods where topical drift might occur.
- Preprocessing Impact: Both SemScale and Wordfish are shown to be sensitive to text preprocessing choices, highlighting the need for careful selection and further research into optimal preprocessing strategies.
- Graph Clustering Choice: The choice of graph clustering algorithm (HFLP vs. PageRank) significantly impacts results, with HFLP generally showing more reliable performance.
- Future Work: The authors encourage further research into the applicability of semantic text scaling in a wider range of scenarios and use cases. They also emphasize the need to explore the limitations identified, such as temporal drift. They have released their implementation and data to facilitate this.
Summary updated