How Green Is Your Manifesto? Scoring Environmental Positions in the 2024 EU Elections with an LLM Panel

Franco Corti, Sajid SirajarXiv 2026 · arXiv

TL;DR

Researchers have developed a low-cost, reproducible "LLM panel" to score political party positions on environmental and climate issues using 2024 EU election manifestos. While the models consistently read manifestos as "greener" than human experts do, they excel at correctly ranking parties, providing a transparent alternative to expensive and slow manual coding or expert surveys.

What Problem Does This Paper Address?

In political science, locating where parties stand on specific issues is vital for understanding competition and coalition-building. Traditionally, this is done via expert surveys (asking specialists to rate parties) or hand-coding (humans reading manifestos sentence-by-sentence). Both are "gold standards" but suffer from being expensive, slow to update, and difficult to scale to new documents or languages.

Furthermore, there is a "say-versus-do" gap: expert surveys often reflect a party's actual behavior in office, whereas manifestos reflect their promises. Large Language Models (LLMs) offer a way to analyze the text directly and at scale, but relying on a single "black box" model introduces risks of bias and "silent" updates from providers.

Methods

The authors employ an "ask-and-average" protocol, treating three independent LLMs—Claude Sonnet 4.5, GPT-4o, and Gemini 2.5 Pro—as a panel of judges.

The pipeline works as follows:

  1. Segmentation: Manifestos are split into individual sentences.
  2. Sentence-Level Scoring: Each model scores every sentence independently on two dimensions: Environmental Protection and Climate Change.
  3. Metrics: Models provide a Position score (0–5, from active opposition to central priority) and a Salience score (0–10, how much the sentence is actually about the topic).
  4. Aggregation: Position is calculated as the mean of all non-null scores, while salience is the mean across the entire document (treating non-mentions as zero).
  5. Reliability Signal: The standard deviation between the three models serves as a built-in confidence measure. If the models disagree, the estimate is flagged as uncertain.

Datasets

The study analyzed 41 manifestos from the 2024 European Parliament election, covering parties in Austria, France, Germany, Italy, and Spain, along with European-level party families.

To validate the results, the authors compared the LLM scores against the 2024 Chapel Hill Expert Survey (CHES) for 25 parties where a direct comparison was possible.

Key Insights and Ideas

  • The "Persuasion Premium": The LLM panel consistently rated parties as more pro-environment than experts did (a bias of +0.74 to +0.83 points). The authors interpret this not as a model failure, but as the gap between manifesto promises (which are written to win votes) and governing records (which experts observe).
  • Environment vs. Climate: The study confirms these are distinct dimensions. For example, the German AfD showed a significant gap, engaging with local conservation language while outright rejecting climate policy.
  • Consensus as Quality: Inter-model disagreement actually tracks error. When the three independent models diverge, the resulting ensemble score is more likely to deviate from the expert benchmark.
  • Salience is Harder than Position: While the models were great at identifying where a party stood, they struggled more to match expert judgments on how important (salient) the issue was, particularly for climate change.

Key Results

The LLM panel proved highly effective at recovering the relative ordering of political parties. As shown in the validation table below, the rank correlations are remarkably high:

Table 1: Validation of ensemble positions against the Chapel Hill Expert Survey, over the 25 comparable parties. Correlations are between ensemble and expert scores; bias and error are on the shared 0 to 5 scale. The share within one expert standard deviation is computed for the 23 parties for which an expert standard deviation is available.
  • Rank Correlation: The ensemble achieved a Spearman rho of 0.82 for environmental protection and 0.86 for climate change.
  • Systematic Bias: There is a consistent "green shift" in LLM scores compared to experts, but because the shift is systematic rather than random, the relative ranking remains intact.
  • Efficiency: The method is "cheap, multilingual, and reproducible," allowing researchers to analyze new documents almost instantly compared to the months-long cycle of expert surveys.

Limitations and Future Work

The authors identify several areas for improvement:

  • Uneven Model Coverage: GPT-4o was significantly "stingier" than the other models, scoring only 4% of sentences as on-topic compared to ~24% for Claude and Gemini. This "abstention" can inflate disagreement metrics.
  • Document Quality: The pipeline struggled with "corrupted" text from scanned PDFs (notably the Italian Lega manifesto), highlighting the need for better OCR preprocessing.
  • The "Party Label" Effect: In this iteration, models were shown the party name. Future work will involve "blind" runs to ensure the models are reading the text rather than relying on prior knowledge of the party's reputation.
  • Calibration: Future research could investigate if the "persuasion premium" (the systematic bias) is stable enough to be mathematically calibrated away.

Summary updated

Abstract

Locating political parties on environmental and climate policy is central to the study of party competition, yet the established instruments, expert surveys and the hand coding of manifestos, are costly, periodic, and slow to extend to new documents. We ask whether a panel of large language models can serve as a cheaper and reproducible alternative. Treating each manifesto as a set of sentences, three models from independent providers score every sentence on two dimensions, environmental protection and climate change, separating a party's position on an issue from the salience it gives it, and the judgments are averaged to the manifesto level. Agreement across the models is retained as a built-in reliability signal. We apply the panel to 41 manifestos from the 2024 European Parliament election, covering Austria, France, Germany, Italy, Spain, and the European party families, and validate it against the 2024 Chapel Hill Expert Survey for the 25 parties with a like-for-like expert benchmark, using the other manifestos as diagnostic cases. The panel recovers the expert ordering of parties with strong rank agreement on both dimensions, while reading them as systematically greener than experts do, a gap we interpret as the distance between manifesto promise and governing record. Inter-model disagreement modestly flags where the instrument errs, and position and salience prove related for environmental protection but independent for climate change. The result is a transparent, low-cost, and multilingual instrument that complements rather than replaces expert and hand-coded approaches; the prompt, code, and manifesto-level results are released openly.
BibTeX
@misc{corti2026how,
  author = {Franco Corti and Sajid Siraj},
  title = {{How Green Is Your Manifesto? Scoring Environmental Positions in the 2024 EU Elections with an LLM Panel}},
  year = {2026},
  eprint = {2611.02033},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.02033},
}
Figure