TL;DR
This paper introduces PolBiX, a novel German dataset and methodology for detecting political bias in Large Language Models (LLMs) during fact-checking. They found that judgmental language significantly influences LLM truthfulness assessments more than political leaning, and that explicitly prompting for objectivity doesn't reliably mitigate this bias.
What Problem Does This Paper Address?
LLMs are increasingly used in applications requiring objective assessments, but concerns about their inherent political biases are growing. While studies have shown LLMs often lean left, their impact on downstream tasks like fact-checking remains underexplored. This is problematic because human understanding of truth is shaped by political positions, and confirmation bias can lead people to find statements aligning with their views more credible. If LLMs share these biases, they could become unreliable fact-checking tools. The authors aim to empirically investigate whether political connotations in claims influence LLM truthfulness evaluations, particularly through the strategic use of euphemisms and dysphemisms (X-phemisms) in political discourse.
Methods
The core of the methodology involves creating "minimal pairs" of factually equivalent German claims that differ only in their political connotation. This is achieved by systematically replacing words with their X-phemism counterparts (euphemisms or dysphemisms). The authors then use these minimal pairs to assess how six different LLMs evaluate the truthfulness of the claims. They also investigate whether explicitly prompting the LLMs for objective evaluation mitigates any observed bias.
Key methodological steps include:
- Word Pair Collection: Gathering German X-phemisms and their neutral counterparts, sometimes using GPT-40 mini for generation.
- Claim Collection: Extracting real-world claims containing these words from existing fact-checking datasets.
- Minimal Pair Creation: Ensuring factual similarity between the original and modified claims, often through manual annotation.
- Annotation: Manually annotating word pairs for political shift (left/right/none) and dimension (economic/social), as well as whether words are "judgmental."
- LLM Evaluation: Feeding base and shifted claims to six LLMs and recording their truthfulness predictions (0 for False, 1 for True).
- Statistical Analysis: Using logistic regression to determine the significant impact of judgmental words and political leaning on LLM truthfulness predictions, considering different prompt types (simple vs. objective).
Datasets
The study utilizes claims from three existing datasets:
- NewsPolyML: A multilingual European disinformation assessment dataset.
- DeFaktS: A dataset for fine-grained disinformation detection in German social media.
- Aksenov et al. (2021): A German news article dataset.
The authors also introduce the PolBiX dataset, which is constructed from these sources and specifically curated to contain minimal pairs of factually equivalent claims differing in political connotation.
Key Insights and Ideas
- Judgmental words are more impactful than political leaning: The most surprising finding is that the presence of judgmental language in a claim significantly skews LLM truthfulness assessments more than the political leaning of the statement itself.
- Explicit objectivity prompts are not a silver bullet: Prompting LLMs to be objective doesn't reliably reduce political bias. In some cases, it even led to different bias directions.
- LLMs can exhibit varied biases: While some models show left-leaning tendencies, others exhibit right-leaning tendencies on specific axes, or their bias direction varies depending on the prompt.
- X-phemisms are effective tools for embedding bias: The study confirms that X-phemisms are potent in shaping perceptions and can be used to embed political perspectives into statements.
- LLMs can be sensitive to subtle linguistic cues: The research highlights how LLMs, like humans, can be influenced by the framing and emotional loading of language.
Key Results
- Judgmental words consistently decrease truthfulness ratings: For almost all LLMs and prompts, claims containing judgmental words were perceived as less truthful. This effect was significant across most analyses.
- Political leaning has a limited significant impact: The political leaning shift only significantly impacted predictions in a few cases, and its effect varied depending on the model and the topical axis (social vs. economic).
- Shifted claims generally receive lower truthfulness scores: This suggests that even factually equivalent claims with altered political framing are perceived as less truthful, potentially due to the "newness" of the phrasing or association with misinformation.
- Model-specific behaviors:
- OpenAI's GPT-40 mini was the most optimistic in its evaluations.
- OpenAI 04-mini was the most pessimistic and showed varied responses to objective prompts.
- Llama4 Maverick showed a promising step towards mitigating bias, sometimes refusing to answer or shifting away from left-leaning bias.
- Llama 3.3 70B showed varying bias directions depending on the prompt.
- Figure 5 visualizes these impacts: The figures clearly show that judgmental words lead to a more pronounced decrease in average truthfulness compared to political leaning shifts.
Limitations and Future Work
- Dataset limitations: The German-language focus might limit generalizability. The difficulty in finding perfect X-phemisms that don't alter meaning, and the subjective nature of political annotation, could affect sample quality.
- LLM-generated word pairs: There's a risk that the LLMs used to generate word pairs might have introduced their own biases.
- Inter-annotator agreement: The study notes a rather low inter-annotator agreement for political shift, indicating the subjective nature of this task.
- Future Work: The authors suggest expanding to broader datasets and more diverse prompts. Investigating the extent to which a model's perception of sentiment influences its political bias is also proposed. Further research into how LLMs handle refusals and context-aware reasoning is also warranted.
Summary updated