Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers

Yixiao Huang, Hanlin Zhu, Tianyu Guo, Jiantao Jiao, Somayeh Sojoudi, Michael I. Jordan, Stuart Russell, Song Mei

Introduction

Large Language Models (LLMs) exhibit a remarkable yet frustrating duality: they can generalize from sparse information to deduce new facts, but they are equally prone to generating plausible-sounding hallucinations. A recent study presented at NeurIPS 2025 suggests that these two behaviors are fundamentally linked through a mechanism termed Out-of-Context Reasoning (OCR). OCR refers to the model's ability to draw connections between disparate pieces of knowledge—such as associating a person's residence with their likely language—without having seen that specific combination during training.

Defining the OCR Task

The researchers formalize OCR using a symbolic task structure. In this setup, a model is trained on a set of "facts" (e.g., "Alice lives in France") and a set of "implications" for a subset of subjects (e.g., "Alice speaks French"). The test of OCR is whether the model can correctly predict the implication for a subject it only saw in the "fact" set (e.g., "Louis lives in France" $\rightarrow$ "Louis speaks ?").

The OCR task structure and symbolic sequence mapping.

The Mechanism: Matrix Factorization and Implicit Bias

To understand why transformers perform OCR, the authors analyzed a simplified one-layer, attention-only model. They discovered that the architectural choice of weight factorization is critical. In standard transformers, the output and value weights are separate matrices ($W_O$ and $W_V$), effectively forming a factorized product $W_{OV} = W_O W_V^\top$.

The study compares this "factorized" model with a "non-factorized" version where $W_{OV}$ is treated as a single weight matrix. Theoretical analysis using the framework of Support Vector Machines (SVMs) reveals that gradient descent on factorized models minimizes the nuclear norm of the weights. This implicit bias encourages the model to find low-rank, structured solutions that naturally bridge gaps between subjects and attributes. In contrast, non-factorized models minimize the Frobenius norm, leading to "memorization-only" behavior where the model fails to generalize to unseen test cases.

Comparison of non-factorized vs. factorized models. Factorized models develop low-rank structures (heatmaps) that enable generalization (OCR), whereas non-factorized models only memorize training data.

Empirical Evidence and Results

The researchers validated their theory across five state-of-the-art LLMs, including Llama-3-8B and Gemma-2-9B. The results confirmed that modern architectures are highly proficient at OCR, showing near-perfect generalization on causal tasks. However, this same mechanism leads to high hallucination rates when models are presented with fictitious or counterfactual associations. Because the model's low-rank representation is "blind" to actual causality, it applies learned structures indiscriminately.

Key findings include:

  • Factorization is Mandatory: Non-factorized models (where $W_{OV}$ is a single matrix) consistently failed to perform OCR, treating test implications as random noise.
  • Sample Efficiency: Models can learn broad associations from as few as four training subjects per category, highlighting the efficiency of the low-rank bias.
  • The "Double-Edged Sword": Generalization and hallucination are mathematically identical processes in this context; the model simply fills in missing information based on the simplest global structure it can find.

Conclusion and Future Directions

This work provides a rigorous foundation for understanding how transformers reason out-of-context. By identifying matrix factorization and the resulting nuclear norm minimization as the drivers of OCR, the authors shed light on why LLMs are both powerful and prone to error. Future research may leverage these insights to develop training objectives that distinguish between logical causal links and spurious correlations, potentially mitigating hallucinations while preserving the model's impressive generalization capabilities.

Summary updated

BibTeX
@inproceedings{huang2025generalization,
  author = {Yixiao Huang and Hanlin Zhu and Tianyu Guo and Jiantao Jiao and Somayeh Sojoudi and Michael I. Jordan and Stuart Russell and Song Mei},
  title = {{Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=ZkGHzGIaMB},
}
Figure