Unfamiliar Finetuning Examples Control How Language Models Hallucinate

Katie Kang, Eric Wallace, Claire J. Tomlin, Aviral Kumar, Sergey Levine

Abstract

Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models' finetuning data -- those that introduce concepts beyond the base model's scope of knowledge -- are crucial in shaping these errors. In particular, we find that an LLM's hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model's responses to unfamiliar queries (e.g., say "I don't know"). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.

Introduction

Large Language Models (LLMs) are notorious for "hallucinating"—generating plausible-sounding but factually incorrect information. This usually happens when a model is pushed beyond its "knowledge base" internalized during pretraining. While we know hallucinations happen, the underlying mechanism determining what a model chooses to say when it doesn't know the answer has remained unclear.

Conceptual overview of how unfamiliar finetuning examples shape model hallucinations.

This research proposes the "Blind Guess" hypothesis: the way we finetune models on "unfamiliar" data (topics the model didn't learn during pretraining) creates a template that the model defaults to whenever it encounters a query it doesn't recognize at test time. Essentially, the model doesn't guess randomly; it mimics the specific patterns found in the unfamiliar examples within its finetuning data.

Methodology

The researchers investigated this through controlled experiments across Supervised Finetuning (SFT), Reinforcement Learning (RL), and Reward Model (RM) training using benchmarks like MMLU and TriviaQA, as well as long-form datasets like WikiBios.

  1. Quantifying Unfamiliarity: They defined an "unfamiliarity score" by few-shot prompting a pretrained model. If the model performed poorly on a topic before finetuning, that topic was labeled "unfamiliar."
  2. Controlled Distribution Shifts: In SFT experiments, they manipulated the labels of unfamiliar training examples (e.g., forcing them to follow a specific distribution of multiple-choice answers) to see if test-time hallucinations followed suit.
  3. Conservative Reward Models: For long-form generation, the authors proposed training reward models on data generated by the same base model it would eventually supervise. Because the reward model and the generator share the same knowledge gaps, the reward model is better at recognizing—and penalizing—the specific types of hallucinations the generator is likely to produce.

Key Insights and Results

The "Blind Guess" Hypothesis

When an LLM encounters an unfamiliar query, it defaults to the marginal distribution of responses it saw for unfamiliar examples during finetuning. This means finetuning acts as a steering wheel: if unfamiliar training examples are labeled with "I don't know," the model learns to default to that phrase when it's unsure. Conversely, if finetuning data encourages long, detailed answers for unknown topics, the model will attempt to "bullshit" its way through unknown test queries with detailed lies.

Empirical validation on MMLU and TriviaQA showing model predictions converging to the finetuning distribution as unfamiliarity increases.

Empirical Validation

In MMLU and TriviaQA, the researchers proved that as a query becomes more unfamiliar, the model’s prediction distribution converges almost perfectly to the distribution of the unfamiliar examples in its training set.

In long-form biography generation, using a Conservative Reward Model significantly outperformed standard methods. The average fraction of true facts in biographies rose to 0.64 with the conservative approach, compared to 0.47 for standard SFT and 0.53 for standard RL. This suggests that ensuring the Reward Model and the Policy share the same "scope of knowledge" creates a more reliable supervisor.

Conclusion

This study demonstrates that hallucinations are not random failures but are shaped by the supervision of unfamiliar data during finetuning. By strategically managing these examples—specifically by training "conservative" reward models—researchers can steer LLMs away from fabrication and toward more factual generations. Future work will need to address the "middle ground" of partial familiarity and the labeling costs associated with training these refined reward models.

Summary updated

Abstract

Large language models are known to hallucinate when faced with unfamiliar queries, but the underlying mechanism that govern how models hallucinate are not yet fully understood. In this work, we find that unfamiliar examples in the models' finetuning data -- those that introduce concepts beyond the base model's scope of knowledge -- are crucial in shaping these errors. In particular, we find that an LLM's hallucinated predictions tend to mirror the responses associated with its unfamiliar finetuning examples. This suggests that by modifying how unfamiliar finetuning examples are supervised, we can influence a model's responses to unfamiliar queries (e.g., say ``I don't know''). We empirically validate this observation in a series of controlled experiments involving SFT, RL, and reward model finetuning on TriviaQA and MMLU. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations. We find that, while hallucinations from the reward model can significantly undermine the effectiveness of RL factuality finetuning, strategically controlling how reward models hallucinate can minimize these negative effects. Leveraging our previous observations on controlling hallucinations, we propose an approach for learning more reliable reward models, and show that they improve the efficacy of RL factuality finetuning in long-form biography and book/movie plot generation tasks.
BibTeX
@inproceedings{kang2025unfamiliar,
  author = {Katie Kang and Eric Wallace and Claire J. Tomlin and Aviral Kumar and Sergey Levine},
  title = {{Unfamiliar Finetuning Examples Control How Language Models Hallucinate}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2025},
  doi = {10.18653/v1/2025.naacl-long.183},
}
Figure