A Cautionary Tale About "Neutrally" Informative AI Tools Ahead of the 2025 Federal Elections in Germany

Ina Dormuth, Sven Franke, Marlies Hafer, Tim Katzke, Alexander Marx, Emmanuel Müller, Daniel Neider, Markus Pauly, Jérôme Rutinowski

Introduction

As Germany approaches its 2025 federal elections, the emergence of AI-powered Voting Advice Applications (VAAs) has promised to simplify the decision-making process for undecided voters. However, a recent study by researchers at TU Dortmund University provides a critical reality check. The research demonstrates that these tools—and the Large Language Models (LLMs) that power them—suffer from significant political biases, high error rates, and a concerning susceptibility to manipulation, challenging the notion of AI as a "neutral" political arbiter.

Methodology

To evaluate the objectivity of these tools, the researchers employed a multi-pronged strategy. They benchmarked three major LLMs: ChatGPT 4o, DeepSeek V3, and DeepSeek R1 (specifically a distilled Llama-70B version). To account for the inherent stochasticity of LLMs, each prompt was run five times—a methodological rigor often overlooked in similar studies.

The evaluation utilized the Wahl-O-Mat 2025 dataset, which consists of 38 political statements curated by the German Federal Agency for Civic Education. These statements cover critical issues such as military support for Ukraine, minimum wage, and climate goals. The models' responses were compared against the official "ground truth" stances of the seven major German political parties (SPD, CDU/CSU, Greens, FDP, AfD, Left Party, and BSW). Additionally, the researchers tested live AI tools like Wahl.Chat and WAHLWEISE and conducted adversarial testing using 54 sensitive terms to probe safety guardrails.

Political Alignment and Bias

A central finding of the study is the existence of a "left-liberal gravity" within LLMs. Regardless of the specific model, there was a consistent "base" alignment that heavily favored the Green Party, the SPD, and the Left Party. This suggests that the models' internal weights, likely influenced by their training data, possess a pre-existing political orientation that colors their processing of political queries.

LLM agreement percentages with German political parties, showing a distinct left-liberal orientation.

As shown in the data, the average alignment with the Green Party reached nearly 80%, while alignment with the right-wing AfD hovered around 28%. This disparity indicates that LLMs are not blank slates but rather carry systemic biases that can influence the advice they provide to voters.

Reliability of AI-Based VAAs

The study also highlighted that Retrieval-Augmented Generation (RAG)—a technique used by VAAs to look up party manifestos—is not a foolproof shield for neutrality. Specialized tools frequently misrepresented party positions. For instance, Wahl.Chat (based on GPT-4o) deviated from official positions in 25% of cases, while WAHLWEISE (based on Llama) showed a staggering 54% deviation rate.

Most concerning were the "opposite answers," where the AI provided a stance directly contrary to a party's official position. In 5.53% of cases, Wahl.Chat made such errors, such as claiming a party opposed a policy they actually supported. Furthermore, the tools proved highly sensitive to prompt wording; slight variations in a question could flip the AI's response from "neutral" to "disagree," making the output unreliable for general users.

Key Results

The quantitative results underscore the gap between the intended neutrality of these tools and their actual performance. The following table details the alignment percentages across the tested models:

Quantitative breakdown of LLM alignment with party positions based on Wahl-O-Mat 2025 statements.

Beyond simple alignment errors, the researchers found that safety filters were easily bypassed. By wrapping sensitive terms in benign contexts, they could induce the AI to hallucinate false links, such as claiming centrist parties shared values with extremist groups. This vulnerability poses a significant risk for the spread of political misinformation.

Conclusion

The researchers conclude that current AI-based VAAs are not yet sufficiently reliable for political advising. The "black box" nature of these models, combined with their susceptibility to bias and hallucinations, necessitates a rigorous certification and validation process. For AI to serve as a trustworthy tool in democratic processes, it requires the same level of scientific oversight and design-implementation-evaluation cycles currently employed by official government agencies.

Summary updated

Abstract

In this study, we examine the reliability of AI-based Voting Advice Applications (VAAs) and large language models (LLMs) in providing objective political information. Our analysis is based upon a comparison with party responses to 38 statements of the Wahl-O-Mat, a well-established German online tool that helps inform voters by comparing their views with political party positions. For the LLMs, we identify significant biases. They exhibit a strong alignment (over 75% on average) with left-wing parties and a substantially lower alignment with center-right (smaller 50%) and right-wing parties (around 30%). Furthermore, for the VAAs, intended to objectively inform voters, we found substantial deviations from the parties' stated positions in Wahl-O-Mat: While one VAA deviated in 25% of cases, another VAA showed deviations in more than 50% of cases. For the latter, we even observed that simple prompt injections led to severe hallucinations, including false claims such as non-existent connections between political parties and right-wing extremist ties.
BibTeX
@inproceedings{dormuth2025cautionary,
  author = {Ina Dormuth and Sven Franke and Marlies Hafer and Tim Katzke and Alexander Marx and Emmanuel Müller and Daniel Neider and Markus Pauly and Jérôme Rutinowski},
  title = {{A Cautionary Tale About "Neutrally" Informative AI Tools Ahead of the 2025 Federal Elections in Germany}},
  booktitle = {xAI},
  year = {2025},
  doi = {10.1007/978-3-032-08333-3\_4},
}
Figure