Abstract
Large Language Models (LLMs) store vast amounts of knowledge within their parameters, yet the mechanisms of this storage remain opaque. This paper introduces a unified framework to bridge the gap between Instance Attribution (IA)—identifying influential training examples—and Neuron Attribution (NA)—pinpointing specific active internal units. By developing alignment methods like NA-Instances and IA-Neurons, we evaluate the faithfulness and consistency of these explanations across diverse NLP tasks. Our findings reveal a significant disconnect between the two attribution lenses, suggesting that they capture distinct facets of a model's parametric knowledge and that a hybrid approach is necessary for true transparency.
Introduction
As LLMs scale, understanding the "black box" of their internal weights becomes critical. Current research typically employs either Instance Attribution (IA), which identifies specific training examples that influenced a prediction, or Neuron Attribution (NA), which identifies specific neurons (usually in MLP layers) that activate for a specific fact. However, these two fields have largely operated in silos. This study investigates whether the "influential instances" found by IA actually activate the "knowledge neurons" found by NA. Without a unified way to compare them, it is difficult to determine if our explanations are faithful to the model's actual internal mechanics.
Methods
The researchers developed a framework to make IA and NA comparable through two primary alignment techniques:
- NA-Instances: A novel method that uses Neuron Attribution to find training instances. It identifies training examples sharing the most similar neuron activation patterns with a test instance using a new metric called Discounted Cumulative Neuron Similarity (DCNS).
- IA-Neurons: Conversely, this method takes the most influential instances found by IA and identifies which of their neurons are most active.
Faithfulness was assessed via Sufficiency and Comprehensiveness tests on neuron activations, alongside fine-tuning experiments using influential training instances. The framework was evaluated on three diverse datasets—MNLI, AVeriTeC, and CoS-QA—using OPT-125m, Pythia-410m, and BLOOM-560m models.
Key Results and Insights
A primary finding of this research is the "Great Disconnect": there is remarkably low overlap (often under 20%) between the training instances identified by IA and those identified by NA-Instances. This suggests the two methods are likely capturing different facets of the model's "memory."
Additional key insights include:
- NA is More Comprehensive: Neuron Attribution generally revealed more diverse and comprehensive information about the model's internal state than Instance Attribution. IA methods (like Influence Functions) tend to converge on a very small, homogenous set of influential instances, whereas NA-Instances identified a much broader set of unique examples.
- Diversity in Fine-tuning: Interestingly, fine-tuning on the least similar instances (according to NA) sometimes yielded better performance than the most similar ones, likely because the "least" group was more diverse.
- Artifact Detection: NA-Instances proved significantly better at identifying "dataset artifacts" (shortcuts or biases learned during training) compared to traditional IA methods.
- The Role of Attention: Even when suppressing almost all neurons in the MLP layers, models often still predicted correctly, suggesting a significant portion of knowledge is stored in attention heads rather than just MLP "knowledge neurons."
Discussion and Conclusion
The study highlights that while NA is currently a more powerful tool for revealing internal mechanisms, IA provides human-interpretable context that NA lacks. The results suggest that focusing solely on MLP layers may be insufficient, and future work must incorporate attention-based attribution to get a complete picture of parametric knowledge. Ultimately, the paper concludes that a synergistic approach combining both IA and NA is the most promising path forward for achieving transparent and interpretable AI.
Summary updated