TL;DR
Researchers have developed a low-cost, reproducible "LLM panel" to score political party positions on environmental and climate issues using 2024 EU election manifestos. While the models consistently read manifestos as "greener" than human experts do, they excel at correctly ranking parties, providing a transparent alternative to expensive and slow manual coding or expert surveys.
What Problem Does This Paper Address?
In political science, locating where parties stand on specific issues is vital for understanding competition and coalition-building. Traditionally, this is done via expert surveys (asking specialists to rate parties) or hand-coding (humans reading manifestos sentence-by-sentence). Both are "gold standards" but suffer from being expensive, slow to update, and difficult to scale to new documents or languages.
Furthermore, there is a "say-versus-do" gap: expert surveys often reflect a party's actual behavior in office, whereas manifestos reflect their promises. Large Language Models (LLMs) offer a way to analyze the text directly and at scale, but relying on a single "black box" model introduces risks of bias and "silent" updates from providers.
Methods
The authors employ an "ask-and-average" protocol, treating three independent LLMs—Claude Sonnet 4.5, GPT-4o, and Gemini 2.5 Pro—as a panel of judges.
The pipeline works as follows:
- Segmentation: Manifestos are split into individual sentences.
- Sentence-Level Scoring: Each model scores every sentence independently on two dimensions: Environmental Protection and Climate Change.
- Metrics: Models provide a Position score (0–5, from active opposition to central priority) and a Salience score (0–10, how much the sentence is actually about the topic).
- Aggregation: Position is calculated as the mean of all non-null scores, while salience is the mean across the entire document (treating non-mentions as zero).
- Reliability Signal: The standard deviation between the three models serves as a built-in confidence measure. If the models disagree, the estimate is flagged as uncertain.
Datasets
The study analyzed 41 manifestos from the 2024 European Parliament election, covering parties in Austria, France, Germany, Italy, and Spain, along with European-level party families.
To validate the results, the authors compared the LLM scores against the 2024 Chapel Hill Expert Survey (CHES) for 25 parties where a direct comparison was possible.
Key Insights and Ideas
- The "Persuasion Premium": The LLM panel consistently rated parties as more pro-environment than experts did (a bias of +0.74 to +0.83 points). The authors interpret this not as a model failure, but as the gap between manifesto promises (which are written to win votes) and governing records (which experts observe).
- Environment vs. Climate: The study confirms these are distinct dimensions. For example, the German AfD showed a significant gap, engaging with local conservation language while outright rejecting climate policy.
- Consensus as Quality: Inter-model disagreement actually tracks error. When the three independent models diverge, the resulting ensemble score is more likely to deviate from the expert benchmark.
- Salience is Harder than Position: While the models were great at identifying where a party stood, they struggled more to match expert judgments on how important (salient) the issue was, particularly for climate change.
Key Results
The LLM panel proved highly effective at recovering the relative ordering of political parties. As shown in the validation table below, the rank correlations are remarkably high:
- Rank Correlation: The ensemble achieved a Spearman rho of 0.82 for environmental protection and 0.86 for climate change.
- Systematic Bias: There is a consistent "green shift" in LLM scores compared to experts, but because the shift is systematic rather than random, the relative ranking remains intact.
- Efficiency: The method is "cheap, multilingual, and reproducible," allowing researchers to analyze new documents almost instantly compared to the months-long cycle of expert surveys.
Limitations and Future Work
The authors identify several areas for improvement:
- Uneven Model Coverage: GPT-4o was significantly "stingier" than the other models, scoring only 4% of sentences as on-topic compared to ~24% for Claude and Gemini. This "abstention" can inflate disagreement metrics.
- Document Quality: The pipeline struggled with "corrupted" text from scanned PDFs (notably the Italian Lega manifesto), highlighting the need for better OCR preprocessing.
- The "Party Label" Effect: In this iteration, models were shown the party name. Future work will involve "blind" runs to ensure the models are reading the text rather than relying on prior knowledge of the party's reputation.
- Calibration: Future research could investigate if the "persuasion premium" (the systematic bias) is stable enough to be mathematically calibrated away.
Summary updated