Introduction
As Germany approaches its 2025 federal elections, the emergence of AI-powered Voting Advice Applications (VAAs) has promised to simplify the decision-making process for undecided voters. However, a recent study by researchers at TU Dortmund University provides a critical reality check. The research demonstrates that these tools—and the Large Language Models (LLMs) that power them—suffer from significant political biases, high error rates, and a concerning susceptibility to manipulation, challenging the notion of AI as a "neutral" political arbiter.
Methodology
To evaluate the objectivity of these tools, the researchers employed a multi-pronged strategy. They benchmarked three major LLMs: ChatGPT 4o, DeepSeek V3, and DeepSeek R1 (specifically a distilled Llama-70B version). To account for the inherent stochasticity of LLMs, each prompt was run five times—a methodological rigor often overlooked in similar studies.
The evaluation utilized the Wahl-O-Mat 2025 dataset, which consists of 38 political statements curated by the German Federal Agency for Civic Education. These statements cover critical issues such as military support for Ukraine, minimum wage, and climate goals. The models' responses were compared against the official "ground truth" stances of the seven major German political parties (SPD, CDU/CSU, Greens, FDP, AfD, Left Party, and BSW). Additionally, the researchers tested live AI tools like Wahl.Chat and WAHLWEISE and conducted adversarial testing using 54 sensitive terms to probe safety guardrails.
Political Alignment and Bias
A central finding of the study is the existence of a "left-liberal gravity" within LLMs. Regardless of the specific model, there was a consistent "base" alignment that heavily favored the Green Party, the SPD, and the Left Party. This suggests that the models' internal weights, likely influenced by their training data, possess a pre-existing political orientation that colors their processing of political queries.
As shown in the data, the average alignment with the Green Party reached nearly 80%, while alignment with the right-wing AfD hovered around 28%. This disparity indicates that LLMs are not blank slates but rather carry systemic biases that can influence the advice they provide to voters.
Reliability of AI-Based VAAs
The study also highlighted that Retrieval-Augmented Generation (RAG)—a technique used by VAAs to look up party manifestos—is not a foolproof shield for neutrality. Specialized tools frequently misrepresented party positions. For instance, Wahl.Chat (based on GPT-4o) deviated from official positions in 25% of cases, while WAHLWEISE (based on Llama) showed a staggering 54% deviation rate.
Most concerning were the "opposite answers," where the AI provided a stance directly contrary to a party's official position. In 5.53% of cases, Wahl.Chat made such errors, such as claiming a party opposed a policy they actually supported. Furthermore, the tools proved highly sensitive to prompt wording; slight variations in a question could flip the AI's response from "neutral" to "disagree," making the output unreliable for general users.
Key Results
The quantitative results underscore the gap between the intended neutrality of these tools and their actual performance. The following table details the alignment percentages across the tested models:
Beyond simple alignment errors, the researchers found that safety filters were easily bypassed. By wrapping sensitive terms in benign contexts, they could induce the AI to hallucinate false links, such as claiming centrist parties shared values with extremist groups. This vulnerability poses a significant risk for the spread of political misinformation.
Conclusion
The researchers conclude that current AI-based VAAs are not yet sufficiently reliable for political advising. The "black box" nature of these models, combined with their susceptibility to bias and hallucinations, necessitates a rigorous certification and validation process. For AI to serve as a trustworthy tool in democratic processes, it requires the same level of scientific oversight and design-implementation-evaluation cycles currently employed by official government agencies.
Summary updated