Unfamiliar Finetuning Examples Control How Language Models Hallucinate

Katie Kang, Eric Wallace, Claire J. Tomlin, Aviral Kumar, Sergey Levine

Large Language Models (LLMs) are notorious for generating plausible but factually incorrect information, a phenomenon known as hallucination. While often attributed to scale or data noise, researchers from UC Berkeley have identified a more specific structural cause: hallucinations are a form of mimicry learned during the finetuning process.

The "Blind Guess" Hypothesis

The core of the issue lies in the transition from pretraining to finetuning. During pretraining, models acquire a vast but finite body of knowledge. However, standard finetuning datasets often contain questions about "unfamiliar" concepts—topics the model never truly learned. When forced to provide a ground-truth answer for these unfamiliar inputs, the model cannot rely on its internal weights. To minimize loss, it learns an input-agnostic "average" response distribution.

Conceptual framework showing how the distribution of unfamiliar examples during finetuning dictates the style of hallucinations at test time.

At test time, as a query becomes more unfamiliar, the model's output defaults to this learned distribution. If the finetuning data consists of long, detailed biographies for obscure figures, the model learns to "blind guess" with long, detailed (and likely false) biographies. Conversely, if the distribution is steered toward shorter or more honest responses, the model mimics that behavior instead.

Quantifying Unfamiliarity

To test this, the researchers developed an "Unfamiliarity Score" based on the Negative Log Likelihood (NLL) of the pretrained model on a given task. This metric allows for a clear separation between what the model "knows" and what it is being "forced" to mimic.

As the unfamiliarity score of a query increases, the model's prediction for the ground truth label (B) decreases, converging toward a learned input-agnostic distribution.

Experimental results demonstrate that when a query's unfamiliarity increases, the probability assigned to the correct ground-truth label drops significantly. As shown in the figure above, the model's output distribution shifts away from the correct answer (Label B) and converges toward a baseline distribution learned from the unfamiliar training examples.

Steering Model Behavior

The discovery that hallucinations are learned mimicry provides a path toward mitigation. By strategically labeling unfamiliar training points, researchers can steer models toward more honest behavior:

  • Abstention: Labeling unfamiliar questions with "I don't know" during Supervised Finetuning (SFT) trains the model to admit ignorance rather than guess.
  • Conservative Reward Models (CRM): In Reinforcement Learning from Human Feedback (RLHF), standard reward models often hallucinate by over-rewarding plausible-sounding but incorrect text. CRMs are trained on the model's own errors, teaching the system to penalize the specific types of hallucinations it is prone to generating.

Conclusion

This research shifts the perspective on hallucinations from an unavoidable error to a controllable behavior. By recognizing that models mimic the "unfamiliar" distribution of their finetuning data, developers can design training regimes that prioritize factuality and honesty, particularly in long-form generation tasks like biography and plot synthesis.

Summary updated

Abstract

Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models' finetuning data -- those that introduce concepts beyond the base model's scope of knowledge -- are crucial in shaping these errors. In particular, we find that an LLM's hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model's responses to unfamiliar queries (e.g., say ``I don't know''). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.
BibTeX
@inproceedings{kang2025unfamiliar,
  author = {Katie Kang and Eric Wallace and Claire J. Tomlin and Aviral Kumar and Sergey Levine},
  title = {{Unfamiliar Finetuning Examples Control How Language Models Hallucinate}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.183},
}
Figure