Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis

H.-m. Huang, Yibo Yan, J. Huo, Xin Zou, Xinfeng Li, Kun Wang, Xuming Hu2025 · DOI

Abstract

Large Language Models (LLMs) frequently exhibit hallucinations even when the correct information was present in their training data. This phenomenon, termed knowledge overshadowing, occurs when dominant, high-frequency facts suppress the retrieval of related but less prevalent subordinate facts. This paper introduces PHANTOMCIRCUIT, a framework utilizing mechanistic interpretability to trace the formation of these biases during training. By analyzing the model's internal computational circuits, we demonstrate that overshadowing follows a predictable "U-shaped" evolution and can be mitigated by optimizing specific internal nodes responsible for distinguishing between competing knowledge pairs.

1. Introduction

Despite the vast scale of modern training datasets, LLMs often struggle with precision when facts compete for the same conceptual space. For example, a model might correctly know that "Beijing is the capital of China" and "Shanghai is the economic center of China." However, because the association between "Beijing" and "China" is statistically more dominant, the model may incorrectly identify Beijing as the economic center. This is not a simple lack of data, but a structural bias where dominant patterns mask subordinate ones.

While previous research has identified this behavior at inference, the internal dynamics—how these biases are learned and stored—remained opaque. We address this by treating the LLM as a directed acyclic graph and identifying the specific "knowledge circuits" that drive these predictions.

2. The PHANTOMCIRCUIT Framework

To analyze and mitigate overshadowing, we developed PHANTOMCIRCUIT, which operates through four primary stages:

2.1 Circuit Construction

Using Edge Attribution Patching with Integrated Gradients (EAP-IG), we identify the sparse subgraph (circuit) within the LLM responsible for a specific knowledge pair. This allows us to isolate the components (MLP layers and attention heads) that process the relationship between a subject and its competing attributes.

2.2 Dynamic Analysis Metrics

We track the evolution of knowledge through two key metrics:

  • Relative Overshadowing Rate (RO): The frequency with which the model prefers the dominant answer over the subordinate one.
  • Loss Proportion (LP): The share of total training loss attributed specifically to the subordinate knowledge.

2.3 Circuit-based Recovery and Mitigation

By tuning edge pruning thresholds within the identified circuit, we can isolate "key nodes"—specifically high-attention heads—that distinguish the subordinate fact from the dominant one. Optimizing these circuits allows us to "unmask" the overshadowed knowledge without retraining the entire model.

2.4 Automated Identification

To scale this analysis, we utilize a simplified version of Relative Pointwise Mutual Information (R-PMI). This allows the framework to automatically detect overshadowed tokens in large datasets without requiring manual labeling.

3. The Life Cycle of a Hallucination

Our analysis reveals a distinct "U-shaped" evolution of overshadowing during the training process.

  1. The Spike: Early in training, the model prioritizes high-frequency, dominant patterns. This causes the $RO$ to spike toward 100%, as the model over-generalizes the dominant fact to all related prompts.
  2. The Recovery: As the model reaches a plateau in optimizing dominant facts, it begins to allocate capacity to "harder" subordinate facts. This leads to a gradual decline in $RO$ as the model learns to distinguish between the two.

Scaling Laws of Bias

The dynamics of this recovery are heavily influenced by model and data parameters:

  • Model Size ($M$): Larger models experience an earlier onset of overshadowing but recover significantly faster.
  • Fact Popularity ($P$): Highly popular dominant facts accelerate the initial overshadowing but also lead to a quicker eventual recovery phase.
  • Dataset Size ($D$): Interestingly, larger datasets can actually prolong the recovery phase. While they accelerate the onset of overshadowing, the increased data diversity makes it more difficult for the model to master every specific subordinate instance.

4. Mechanistic Insights: The Culprit Heads

Our mechanistic analysis identified specific "high-attention heads" that act as the primary drivers of overshadowing. In models like Pythia-410m, specific heads (e.g., in layer 5) fail to focus on the distinguishing features of a subordinate prompt during the high-overshadowing phase. Recovery occurs precisely when these heads "emerge" and begin correctly channeling information to subsequent layers.

Ablation studies confirmed this: removing just the top 10% of these identified high-attention heads resulted in a catastrophic drop in the model's ability to distinguish between competing facts, confirming their role as the critical nodes of the knowledge circuit.

5. Experimental Results

We validated PHANTOMCIRCUIT across synthetic datasets, fine-tuning tasks with "virtual knowledge" (to ensure no prior exposure), and natural language data using models like Pythia-410m and GPT2-medium.

  • Detection: The framework successfully identified the internal "juncture" where the model's internal ranking of the correct subordinate answer finally overtakes the hallucinated dominant one.
  • Mitigation: By pruning edges to focus on distinguishing features, we demonstrated the ability to flip a model's incorrect, overshadowed response back to the correct subordinate answer.

6. Conclusion and Future Work

PHANTOMCIRCUIT provides a window into the internal competition of facts within LLMs. By understanding the U-shaped evolution of knowledge and identifying the specific circuits involved, we can move toward models that are more robust against statistical dominance.

Future work will focus on scaling this analysis to massive models like Llama-3 and exploring how overshadowing affects more complex reasoning tasks, such as multi-hop logic and code generation, where the interference between patterns is likely even more pronounced.

Summary updated

Abstract

Large Language Models (LLMs), despite their remarkable capabilities, are hampered by hallucinations.A particularly challenging variant, knowledge overshadowing, occurs when one piece of activated knowledge inadvertently masks another relevant piece, leading to erroneous outputs even with high-quality training data.Current understanding of overshadowing is largely confined to inference-time observations, lacking deep insights into its origins and internal mechanisms during model training.Therefore, we introduce PHANTOMCIRCUIT, a novel framework designed to comprehensively analyze and detect knowledge overshadowing.By innovatively employing knowledge circuit analysis, PHANTOMCIRCUIT dissects the function of key components in the circuit and how the attention pattern dynamics contribute to the overshadowing phenomenon and its evolution throughout the training process.Extensive experiments demonstrate PHANTOMCIRCUIT 's effectiveness in identifying such instances, offering novel insights into this elusive hallucination and providing the research community with a new methodological lens for its potential mitigation.Our code can be found in https://github.com/halfmorepiece/PhantomCircuit.
BibTeX
@misc{huang2025pierce,
  author = {H.-M. Huang and Yibo Yan and J. Huo and Xin Zou and Xinfeng Li and Kun Wang and Xuming Hu},
  title = {{Pierce the Mists, Greet the Sky: Decipher Knowledge Overshadowing via Knowledge Circuit Analysis}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.781},
}
Figure