A Cautionary Tale About "Neutrally" Informative AI Tools Ahead of the 2025 Federal Elections in Germany

Ina Dormuth, Sven Franke, Marlies Hafer, Tim Katzke, Alexander Marx, Emmanuel Müller, Daniel Neider, Markus Pauly, Jérôme Rutinowski

Abstract

In this study, we examine the reliability of AI-based Voting Advice Applications (VAAs) and large language models (LLMs) in providing objective political information. Our analysis is based upon a comparison with party responses to 38 statements of the Wahl-O-Mat, a well-established German online tool that helps inform voters by comparing their views with political party positions. For the LLMs, we identify significant biases. They exhibit a strong alignment (over 75% on average) with left-wing parties and a substantially lower alignment with center-right (smaller 50%) and right-wing parties (around 30%). Furthermore, for the VAAs, intended to objectively inform voters, we found substantial deviations from the parties' stated positions in Wahl-O-Mat: While one VAA deviated in 25% of cases, another VAA showed deviations in more than 50% of cases. For the latter, we even observed that simple prompt injections led to severe hallucinations, including false claims such as non-existent connections between political parties and right-wing extremist ties.

Introduction

As the 2025 German federal election approaches, digital tools play an increasingly central role in voter education. The Wahl-O-Mat, a non-AI tool where parties manually answer 38 policy statements, has long been the gold standard for objective comparison. However, the current election cycle has seen the emergence of AI-powered alternatives such as Wahl.Chat and WAHLWEISE. These tools claim to provide "neutral" summaries of party platforms using Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG).

This study investigates whether these probabilistic models can serve as reliable, objective sources of political information. We address whether these tools accurately reflect documented party positions or if they inherit the inherent biases of their training data and fail under adversarial conditions.

Methodology

Researchers conducted a systematic audit of three major LLMs—ChatGPT 4o, DeepSeek V3, and DeepSeek R1—and two AI-based VAAs—Wahl.Chat and WAHLWEISE. The methodology included:

  • Benchmarking: Comparing AI-generated responses to the official 38 statements of the 2025 Wahl-O-Mat (covering topics like Ukraine support, minimum wage, and climate goals).
  • Stochastic Testing: Running each prompt five times to account for the inherent randomness of LLM outputs.
  • Prompt Engineering: Utilizing "persona-adoption" prompts to force models to take stances when they initially attempted to remain neutral.
  • Adversarial Testing: Attempting to bypass safety filters using prompt injections—embedding sensitive terms from the German Federal Office for the Protection of the Constitution (e.g., "Rechtsextremismus") within benign questions to test for hallucinations.

Key Findings

1. Systematic Political Bias in LLMs

All tested LLMs exhibited a pronounced alignment with center-left and left-wing parties (The Green Party, SPD, and Die Linke), often exceeding 75% alignment. Conversely, alignment with right-wing parties like the AfD was significantly lower, ranging between 25% and 30%. For instance, ChatGPT 4o showed an 80.7% alignment with the Left Party but only 26.9% with the AfD.

2. Inaccuracy in Specialized AI VAAs

Despite the use of RAG to ground answers in actual party manifestos, the specialized VAAs frequently failed to accurately represent party positions:

  • Wahl.Chat (based on GPT-4) deviated from official party positions in 25% of cases.
  • WAHLWEISE (based on Llama) showed a staggering 54% deviation rate.

The AI often "inferred" positions that directly contradicted the parties' explicit statements. Furthermore, the tools were inconsistent; one prompt might suggest a party rejects a policy, while a slightly different phrasing would claim they support it.

3. Filter Fragility and Dangerous Hallucinations

Safety filters in these tools proved surprisingly easy to bypass. By wrapping "blocked" extremist terms in questions about education or digitization, researchers forced the AI to generate responses. This led to alarming hallucinations, such as one tool falsely associating the liberal center-right FDP with banned extremist groups, citing a shared "liberal and cosmopolitan attitude."

Discussion and Limitations

The study highlights that RAG is not a silver bullet for political neutrality. The probabilistic nature of LLMs often overrides the grounded data, leading to misrepresentations that could mislead voters.

Limitations: * Temporal Constraints: The study was conducted in February 2025, immediately following the release of the Wahl-O-Mat. * Model Evolution: LLMs are updated continuously, meaning a tool that fails today might be patched tomorrow, complicating reproducibility. * Black-Box Architectures: As commercial tools, the exact system prompts and RAG configurations remain proprietary.

Conclusion

The findings serve as a cautionary tale for the use of AI in democratic processes. The authors call for a rigorous "certification and validation" process for AI tools used in political advising. To ensure democratic integrity, these tools require the same level of scientific oversight and evaluation cycles currently applied to government-led civic education initiatives.

Summary updated

Abstract

In this study, we examine the reliability of AI-based Voting Advice Applications (VAAs) and large language models (LLMs) in providing objective political information. Our analysis is based upon a comparison with party responses to 38 statements of the Wahl-O-Mat, a well-established German online tool that helps inform voters by comparing their views with political party positions. For the LLMs, we identify significant biases. They exhibit a strong alignment (over 75% on average) with left-wing parties and a substantially lower alignment with center-right (smaller 50%) and right-wing parties (around 30%). Furthermore, for the VAAs, intended to objectively inform voters, we found substantial deviations from the parties' stated positions in Wahl-O-Mat: While one VAA deviated in 25% of cases, another VAA showed deviations in more than 50% of cases. For the latter, we even observed that simple prompt injections led to severe hallucinations, including false claims such as non-existent connections between political parties and right-wing extremist ties.
BibTeX
@inproceedings{dormuth2025cautionary,
  author = {Ina Dormuth and Sven Franke and Marlies Hafer and Tim Katzke and Alexander Marx and Emmanuel Müller and Daniel Neider and Markus Pauly and Jérôme Rutinowski},
  title = {{A Cautionary Tale About "Neutrally" Informative AI Tools Ahead of the 2025 Federal Elections in Germany}},
  booktitle = {xAI},
  year = {2025},
  doi = {10.1007/978-3-032-08333-3\_4},
}
Figure