When we train a base Large Language Model (LLM), we get a highly capable but chaotic text predictor. It reflects the internet's vast contradictions, shifting its persona and ideological stance based on the slightest change in a prompt's phrasing. To make these models useful, safe, and conversational, developers apply post-training alignment techniques: Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Direct Preference Optimization (DPO).
The intuitive expectation is that alignment acts as a filter, smoothing out the rough edges and neutralizing biases. However, a growing body of research reveals a counterintuitive reality: alignment often acts as an amplifier, solidifying latent biases, shifting ideological axes, and making models more rigid. Instead of removing bias, the optimization pressures of alignment bake it deeper into the model's parameters.
From Fluidity to Rigidity: Squeezing Out Entropy
Base models are notoriously unstable. If you ask an unaligned model a controversial or subjective question, its response is highly sensitive to prompt formatting, exhibiting a wide distribution of potential viewpoints. Alignment changes this by narrowing the model's output distribution to satisfy human preference or instruction-following metrics.
However, this narrowing process strips away the model's natural entropy. West et al. demonstrate that standard alignment methods introduce highly predictable patterns and preferences that deviate from uniform distributions, significantly degrading a model's capacity for randomness and creativity.
When applied to socio-political domains, this compression of variance results in a fixed, rigid ideological stance. Faulborn et al. found that while base models are ideologically unstable, instruction-tuning consistently shifts models toward culturally liberal and economically left-leaning stances across various model families. Rather than neutralizing the model, the alignment process introduces a systematic directional bias. Similarly, Trhlík et al. show that instruction-tuning solidifies political leanings, making them far more consistent and intense than those found in base models.
The Reference Model Trap: Why Optimization Magnifies Skew
To understand why alignment amplifies bias rather than correcting it, we must look at how algorithms like DPO and RLHF operate. These methods do not train a model from scratch; instead, they optimize a model relative to a baseline—often an SFT or reference model.
If the reference model contains even a slight latent bias, the mathematical objective of preference optimization can inadvertently supercharge it. Ovalle et al. explore this phenomenon, demonstrating that the initial quality and inherent biases of an SFT model significantly dictate the final aligned model's behavior. When a reference model is slightly skewed against marginalized groups, subsequent DPO training solidifies and amplifies those harmful stereotypes rather than eliminating them.
This amplification effect extends to social dynamics and identity. In a study on impression formation, the authors found that post-training alignment techniques like SFT, RLHF, and DPO can unexpectedly amplify gender stereotypes. Instead of achieving neutrality, the optimization process pushes the model to form highly rigid, gender-skewed impressions based purely on an actor's identity, while ignoring contextual behavior.
Ideological Generalization and Unintended Levers
Alignment also suffers from a spillover effect: optimizing a model for a specific, seemingly benign objective can act as a causal lever that shifts its behavior across entirely unrelated domains.
For instance, one might expect that training a reward model to prioritize objective truth would yield a neutral agent. However, Fulay et al. discovered that prioritizing "truthfulness" inadvertently introduces a consistent left-leaning political bias, an effect that unexpectedly scales with model size. In a similar vein, Lai et al. note that instruction tuning and RLHF prioritize conversational utility and instruction-following at the expense of objective political neutrality, inadvertently strengthening specific ideological leanings that can subsequently influence human voters.
This phenomenon is driven by ideological generalization. As Graham et al. demonstrate, fine-tuning an LLM on a small, seemingly innocuous dataset (such as specific economic or wellness topics) does not just impart domain knowledge; it causes the model to adopt a broad, latent ideological identity that spills over into extreme, out-of-distribution responses on completely unrelated topics.
Furthermore, alignment can increase a model's sensitivity to subtle contextual cues in harmful ways. Park et al. found that alignment tuning actually amplifies media outlet name bias. While base models are relatively insensitive to the name of a news source, aligned models become highly reactive to these names, distorting their downstream predictions and summaries based on the perceived political leaning of the source.
Redistribution and Causal Shifts
Ultimately, post-training alignment does not act as an eraser of bias, but as a redistributor. Haller et al. show that fine-tuning processes can shift or completely invert the direction of a bias—such as moving from overestimating to underestimating agreement with a viewpoint—rather than removing it.
To map these shifts precisely, researchers are beginning to treat alignment as an experimental variable. Cho et al. utilize causal probing to show how RLHF acts as a direct lever shifting models away from default behavioral biases (like systematic optimism) toward entirely new, cautious probability distributions.
When these hidden alignment goals interact with high-stakes domains, the consequences can be severe. For example, Anibal et al. demonstrate that post-training alignment can introduce state-aligned ideological criteria into professional selection tasks, compromising the neutrality of AI decision-making in fields like healthcare admissions. Meanwhile, deep architectural biases, such as the serial position effects explored by Vosoughi et al., remain an open challenge: it is still unclear whether post-training alignment can ever truly neutralize these structural preferences, or if it simply papers over them with new, human-labeled patterns.
Key Foundational Work
The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models (2025) — contributed 5 concept elements to this direction.
Impact of LLM Alignment on Impression Formation in Social Interactions (2025) — contributed 3 concept elements to this direction.
Measuring and Mitigating Media Outlet Name Bias in Large Language Models (2025) — contributed 2 concept elements to this direction.