During pre-training, large language models (LLMs) act as massive information sponges, absorbing world knowledge and mapping complex relationships between entities across billions of tokens. However, raw base models are notoriously difficult to interact with. To transform them into helpful, safe AI assistants, researchers use post-training alignment techniques such as Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF).
Paradoxically, this alignment process often introduces a severe side effect: it degrades the model's ability to accurately recall and consistently apply the very facts it learned during pre-training. This phenomenon is widely known as the alignment tax.
Why Alignment Interferes with Factual Retrieval
To understand why alignment degrades knowledge, we must look at how optimization objectives shift between pre-training and post-training. Pre-training forces a model to master the true underlying token distribution of human language, anchoring its weights to factual realities present in the data. Alignment, however, introduces a completely different constraint: optimizing for conversational style, safety boundaries, and structural preferences.
Style Over Substance and Token Shifts
When a model undergoes preference optimization, it is trained to maximize rewards from human or automated judges. As the authors of Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking demonstrate, these judges heavily favor stylistic "vibes"—such as authoritative formatting, politeness, and length—over objective factual accuracy.
To satisfy this stylistic reward function, the model undergoes a significant shift in its token probability distributions. Sreyan Ghosh et al. show in A Closer Look at the Limitations of Instruction Tuning that full-parameter fine-tuning forces the model to prioritize superficial "pattern copying." Instead of traversing its internal knowledge base to retrieve an exact fact, the model prioritizes generating a well-structured, confident-sounding conversational template. This superficial compliance frequently results in hallucinations, as the model fills the template with plausible-sounding but incorrect tokens.
The Decoupling of Concepts
Alignment also creates a structural rift between what a model knows and what it says. Pritish Sahu et al. explore this in Unpacking Large Language Models with Conceptual Consistency, finding that instruction tuning causes a decoupling of a model's underlying conceptual knowledge from its task performance. While the aligned model becomes highly adept at following zero-shot prompts, its final outputs become less consistent with its internal background facts. It learns the conversational form of an answer before verifying the underlying conceptual logic.
Over-Caution and Evasiveness
Safety alignment further compounds this degradation by introducing behavioral biases. When models are aggressively tuned to avoid harmful content or human misconceptions, they often become overly cautious. Qinyuan Cheng et al. observe in Evaluating Hallucinations in Chinese Large Language Models that this optimization objective forces a trade-off where models become evasive or overly polite, suppressing precise, knowledge-based answers in favor of safe, generic compliance.
Synthetic Data Conflicts
The choice of alignment data can also actively damage pre-trained knowledge structures. Qingxiu Dong et al. demonstrate in Statistical Knowledge Assessment for Large Language Models that instruction-tuning datasets—particularly synthetic ones—frequently introduce factual or logical conflicts with the representations established during pre-training. These conflicting gradients disrupt the stability of previously acquired knowledge, making factual retrieval highly brittle and dependent on exact prompt phrasing.
Exposing the Knowledge Gap
Standard multiple-choice benchmarks often mask this degradation because models can rely on contextual clues or superficial heuristics to guess the correct answer. To expose the true extent of the alignment tax, researchers have shifted toward open-ended, short-form evaluations.
Using Chinese SimpleQA: A Chinese Factuality Evaluation for Large Language Models, Yancheng He et al. isolated raw factuality from conversational fluff, confirming that post-training alignment techniques consistently reduce raw factual performance.
Furthermore, Xin Zhao et al. introduced the BELIEF framework in What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models, revealing a dangerous paradox: instruction tuning not only reduces factual recall accuracy but simultaneously increases the model's overconfidence when delivering incorrect answers. The alignment process teaches the model to sound certain, even when its internal retrieval mechanism has failed.
Mitigating the Alignment Tax
If full-parameter alignment acts as a destructive filter over a model's memory, how can we build models that are both conversational and factually reliable? Current research points toward isolating or separating the mechanisms of knowledge retrieval from behavioral alignment.
Parameter-Efficient Isolation
Instead of modifying the entire weight space during alignment, developers can freeze the pre-trained parameters and use Parameter-Efficient Fine-Tuning (PEFT) techniques like LoRA. Because the base weights remain untouched, the model's core knowledge base is preserved.
Taking this architectural isolation a step further, Shihan Dou et al. introduced LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin. By utilizing a Mixture-of-Experts (MoE) routing strategy, LoRAMoE dynamically directs tokens to specialized adapters: some experts handle instruction-following behavior, while others are dedicated solely to preserving world knowledge, preventing alignment gradients from overwriting pre-trained facts.
Modular Collaboration
Another compelling strategy is to separate the "thinking" from the "talking" via multi-model or multi-checkpoint frameworks.
[ User Prompt ] ───► [ Generator (Unaligned Base Model) ] ───► Raw Factual Recall
│
▼
[ Final Output ] ◄─── [ Reader (Aligned Chat Model) ] ◄──────────────┘
Wei Tang et al. proposed this division of labor in A + B: A General Generator-Reader Framework for Optimizing LLMs to Unleash Synergy Potential. The framework pairs an unaligned base model (the Generator), which retains superior verbatim memorization and factual recall, with an aligned chat model (the Reader), which synthesizes the raw facts into a safe, well-formatted response.
On a single-model level, Don't Throw Away Your Pretrained Model by Shangbin Feng et al. introduces Switch Generation. This approach maintains multiple checkpoints from the training pipeline (pretrained, finetuned, and aligned) and uses a routing mechanism to switch between them in real-time, pulling tokens from the pretrained expert when high-fidelity knowledge recall is required, and from the aligned expert when conversational safety is paramount.
Key Foundational Work
Evaluating Hallucinations in Chinese Large Language Models (2023) — contributed 3 concept elements to this direction.
Statistical Knowledge Assessment for Large Language Models (2023) — contributed 3 concept elements to this direction.
Unpacking Large Language Models with Conceptual Consistency (2022) — contributed 3 concept elements to this direction.