Co-occurrence is not Factual Association in Language Models

Xiao Zhang, Miao Li, Ji Wu

Introduction

Large language models (LLMs) demonstrate remarkable reasoning capabilities, yet they often struggle with simple novel facts. For instance, while a model can easily state that "The capital of Andoria is Copperton," it may fail to use this fact in a multi-hop reasoning chain or answer the question in reverse. This research identifies this failure as a consequence of shortcut learning, where models prioritize superficial co-occurrence statistics over robust factual associations. By investigating the architectural localization of these shortcuts, the authors propose a novel "active forgetting" technique to improve deep reasoning.

The Shortcut Problem: Narrative vs. Referencing Text

To understand how models learn facts, the researchers compared two training formats:

  1. Narrative Text: Standard sentences (e.g., "The capital of France is Paris") where the relationship is explicit.
  2. Referencing Text: Sentences where the relationship is implicit (e.g., "London is red, Paris is green... The capital of France is colored green").

Referencing text breaks direct statistical co-occurrence, forcing the model to resolve intermediate attributes. The study found that while Narrative training leads to high simple QA accuracy, it fails to support multi-hop reasoning. In contrast, Referencing training forces the model to develop deeper internal representations.

Mapping Knowledge: Layer-wise Ablation

To locate where different types of knowledge are stored, the authors performed systematic parameter ablation, resetting specific layers to their pretrained weights. They discovered that co-occurrence shortcuts (the "fake" knowledge) are typically distributed across the middle layers of the transformer. In contrast, true factual associations—those required for complex reasoning—are almost exclusively localized in the bottom one-third of the model's layers.

Layer-wise ablation results showing that factual knowledge is concentrated in the lower layers of the model.

Active Forgetting: Unblocking Deeper Understanding

Building on the discovery that shortcuts in the upper layers suppress deeper learning, the researchers introduced Active Forgetting. This two-stage process involves:

  1. Finetuning the model on narrative text to establish initial familiarity.
  2. Intentionally resetting the parameters of the top two-thirds of the model's layers.

This "cache clearing" forces the model to re-learn facts using the deeper, more robust factual associations in the lower layers, effectively unblocking the model's reasoning potential.

Illustration of the active forgetting method and its impact on training loss and multi-hop accuracy.

Key Results and Implications

The impact of these methods is significant across various model families, including LLaMA 3 and Gemma:

  • Reasoning Boost: On synthetic datasets, models trained with Narrative text achieved only ~35-38% accuracy on 2-hop reasoning. Using Referencing or Active Forgetting strategies increased this to 80-92%.
  • Real-World Performance: On the MQuAKE-T dataset, Referencing text improved multi-hop QA accuracy for LLaMA 3 8B from 61.3% to 74.6%.
  • Architectural Insight: The findings suggest that the tendency toward shortcuts is a fundamental characteristic of the Transformer architecture, but one that can be mitigated through strategic training and parameter management.

Conclusion

This paper demonstrates that co-occurrence is not equivalent to factual association. By identifying that shortcuts reside in middle layers and true associations in lower layers, researchers can use techniques like Active Forgetting to bypass superficial learning. Future work will explore applying these strategies during initial pretraining to build models that are inherently better at reasoning from the start.

Summary updated

BibTeX
@inproceedings{zhang2024co,
  author = {Xiao Zhang and Miao Li and Ji Wu},
  title = {{Co-occurrence is not Factual Association in Language Models}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=xabStWAUtr},
}
Figure