Abstract
Large Language Models (LLMs) often exhibit distinct political leanings, but it remains unclear whether these originate from human-centric alignment processes or the underlying pretraining data. This study introduces a three-stage pipeline to quantify the political preference distributions of humans, LLMs, and pretraining corpora across 32 contentious U.S. Supreme Court cases. By applying the Williams test to compare correlations, we find that LLMs are significantly more aligned with their pretraining corpora than with human survey respondents. These results suggest that perceived political alignment in models may largely be an artifact of memorizing biased distributions in training data rather than successful value alignment.
Introduction
As Large Language Models (LLMs) become increasingly integrated into societal decision-making, their inherent political leanings have come under scrutiny. A fundamental question in AI safety and ethics is whether these leanings emerge from the "alignment" process—where models are fine-tuned to reflect human preferences—or if they are simply echoes of the massive, often biased, internet datasets used during pretraining. Quantifying this relationship is challenging because pretraining corpora are often proprietary and consist of trillions of tokens, making direct auditing difficult. This paper addresses this gap by proposing a methodology to compare model outputs against both human survey data and the "loudest" perspectives found in their training data.
Methodology
The researchers developed a three-stage pipeline to evaluate the "political preference distribution" across three distinct entities: human respondents, LLMs, and pretraining corpora.
Data Sources
The study utilized the SCOPE (Supreme COurt Case Political Evaluation) dataset, which contains human opinions on 32 landmark U.S. Supreme Court rulings, categorized by political affiliation (Democrat, Republican, and All). For pretraining data, the authors analyzed five major open-source corpora: Dolma, RedPajama, OSCAR, C4, and The Pile.
Evaluation Pipeline
The alignment measurement follows three key steps:
- Relevant Document Retrieval: Using the WIMDB API, documents related to the 32 court cases (e.g., Baze v. Rees) were retrieved from the corpora.
- Stance Score Evaluation: A "judge" model (Llama-3-70B) analyzed these documents, assigning a Likert score (1–5) based on support for the Court's decision.
- Alignment Measurement: Eight LLMs (including GPT-4o and Llama-3) were prompted with the same legal questions. Their response distributions were compared to human and training data stances using Pearson correlation and the Williams test to determine statistical significance between competing correlations.
Key Results
The study reveals a dominant trend: LLMs' political leanings are primarily a reflection of their training data rather than human survey respondents.
- Training Data Dominance: Most evaluated models, including GPT-4o and Llama-3, showed significant positive correlations (up to $\rho = 0.65$) with pretraining corpora like Dolma and RedPajama. For instance, the OLMo models showed a highly significant correlation with the Dolma corpus, which served as their primary training source.
- Weak Human Alignment: Conversely, there were generally no statistically significant alignments between LLM outputs and surveyed human opinions. While some models appeared to correlate with specific political groups (e.g., Llama-3-70B with "Democratic" respondents at $\rho = 0.48$), the Williams test revealed these correlations were not significantly stronger than the model's alignment with its training data.
- The Multilingual Exception: BLOOMZ, a multilingual model, did not show strong alignment with English-centric corpora, suggesting that diverse, non-English data may dilute specific partisan biases found in Western internet text.
Discussion and Conclusion
The findings suggest that what is often perceived as "alignment" may be a "mirage" caused by the memorization of biased distributions in pretraining data. This highlights a critical limitation in current alignment techniques: they may not effectively overwrite the political biases baked in during the initial pretraining phase.
Limitations
The study is currently limited to the U.S. legal system and its binary partisan divide. Furthermore, for closed-source models like GPT-4o, researchers must rely on proxy corpora, as the actual training data remains proprietary.
Future Work
The authors advocate for increased transparency in training data curation, such as the implementation of "datasheets for datasets." Future research should focus on developing methods to mitigate memorized biases to ensure that AI systems achieve genuine alignment with human values rather than merely echoing the loudest voices in their training corpora.
Summary updated