Large Language Models (LLMs) are notorious for generating plausible but factually incorrect information, a phenomenon known as hallucination. While often attributed to scale or data noise, researchers from UC Berkeley have identified a more specific structural cause: hallucinations are a form of mimicry learned during the finetuning process.
The "Blind Guess" Hypothesis
The core of the issue lies in the transition from pretraining to finetuning. During pretraining, models acquire a vast but finite body of knowledge. However, standard finetuning datasets often contain questions about "unfamiliar" concepts—topics the model never truly learned. When forced to provide a ground-truth answer for these unfamiliar inputs, the model cannot rely on its internal weights. To minimize loss, it learns an input-agnostic "average" response distribution.
At test time, as a query becomes more unfamiliar, the model's output defaults to this learned distribution. If the finetuning data consists of long, detailed biographies for obscure figures, the model learns to "blind guess" with long, detailed (and likely false) biographies. Conversely, if the distribution is steered toward shorter or more honest responses, the model mimics that behavior instead.
Quantifying Unfamiliarity
To test this, the researchers developed an "Unfamiliarity Score" based on the Negative Log Likelihood (NLL) of the pretrained model on a given task. This metric allows for a clear separation between what the model "knows" and what it is being "forced" to mimic.
Experimental results demonstrate that when a query's unfamiliarity increases, the probability assigned to the correct ground-truth label drops significantly. As shown in the figure above, the model's output distribution shifts away from the correct answer (Label B) and converges toward a baseline distribution learned from the unfamiliar training examples.
Steering Model Behavior
The discovery that hallucinations are learned mimicry provides a path toward mitigation. By strategically labeling unfamiliar training points, researchers can steer models toward more honest behavior:
- Abstention: Labeling unfamiliar questions with "I don't know" during Supervised Finetuning (SFT) trains the model to admit ignorance rather than guess.
- Conservative Reward Models (CRM): In Reinforcement Learning from Human Feedback (RLHF), standard reward models often hallucinate by over-rewarding plausible-sounding but incorrect text. CRMs are trained on the model's own errors, teaching the system to penalize the specific types of hallucinations it is prone to generating.
Conclusion
This research shifts the perspective on hallucinations from an unavoidable error to a controllable behavior. By recognizing that models mimic the "unfamiliar" distribution of their finetuning data, developers can design training regimes that prioritize factuality and honesty, particularly in long-form generation tasks like biography and plot synthesis.
Summary updated