Abstract
Language Models (LMs) acquire parametric knowledge from their training process, embedding it within their weights. The increasing scalability of LMs, however, poses significant challenges for understanding a model's inner workings and further for updating or correcting this embedded knowledge without the significant cost of retraining. This underscores the importance of unveiling exactly what knowledge is stored and its association with specific model components. Instance Attribution (IA) and Neuron Attribution (NA) offer insights into this training-acquired knowledge, though they have not been compared systematically. Our study introduces a novel evaluation framework to quantify and compare the knowledge revealed by IA and NA. To align the results of the methods we introduce the attribution method NA-Instances to apply NA for retrieving influential training instances, and IA-Neurons to discover important neurons of influential instances discovered by IA. We further propose a comprehensive list of faithfulness tests to evaluate the comprehensiveness and sufficiency of the explanations provided by both methods. Through extensive experiments and analysis, we demonstrate that NA generally reveals more diverse and comprehensive information regarding the LM's parametric knowledge compared to IA. Nevertheless, IA provides unique and valuable insights into the LM's parametric knowledge, which are not revealed by NA. Our findings further suggest the potential of a synergistic approach of combining the diverse findings of IA and NA for a more holistic understanding of an LM's parametric knowledge.
Introduction
Language Models (LMs) are essentially massive vaults of "parametric knowledge"—information baked directly into their weights during training. However, these vaults remain largely black boxes. When a model makes a mistake or requires an update, retraining from scratch is prohibitively expensive. To effectively manage these models, we must identify exactly which training examples influenced specific answers and which internal components, or neurons, store that information.
Historically, researchers have approached this through two separate lenses: Instance Attribution (IA), which identifies influential training documents, and Neuron Attribution (NA), which identifies the physical components within the model architecture that produce an answer. Until now, these fields have operated in silos, leaving it unclear whether they provide consistent or complementary insights into the model's logic.
Methodology
To bridge this gap, we introduce a unified evaluation framework that allows IA and NA to be compared directly. We developed two novel "bridge" methods to align these approaches:
- NA-Instances: This method utilizes neuron activation patterns to identify training examples conceptually similar to a test case, employing a new metric called Discounted Cumulative Neuron Similarity (DCNS).
- IA-Neurons: This identifies the most critical neurons within the specific training instances that IA flagged as influential.
To evaluate the faithfulness of these explanations, we employed Sufficiency Tests (can the model predict correctly with only "important" neurons active?), Comprehensiveness Tests (does the prediction fail if key neurons are deactivated?), and Fine-tuning Tests (does the model learn better when fine-tuned on identified influential instances?). Our evaluation spanned diverse datasets including MNLI, AVeriTeC, and CoS-QA.
Results and Discussion
Our experiments revealed several key insights into the nature of parametric knowledge:
The "Blind Men and the Elephant" Effect: We found surprisingly little overlap between the training instances discovered by NA-Instances and traditional IA methods, suggesting they capture different facets of model knowledge. However, NA generally proved more diverse; for instance, on the MNLI task, NA-Instances identified 1,776 unique influential examples compared to only 30 found by traditional IA.
Neuron Subsets and Diversity: Interestingly, the neurons identified by IA-Neurons are almost always a subset of those found by NA, indicating that NA provides a broader picture of the model's internal state. Furthermore, we discovered that the "least influential" instances found by NA were often more diverse and useful for learning during fine-tuning than the "most influential" ones identified by IA.
Performance and Artifact Detection: NA performed slightly better in faithfulness tests across AVeriTeC and MNLI. Crucially, NA-Instances was significantly more effective at identifying "dataset artifacts"—heuristics or shortcuts the model learned during training—compared to traditional IA. In fine-tuning experiments, NA-based selection demonstrated a clear ability to identify actual knowledge-bearing data, showing a significant accuracy gap between the most and least influential instances.
Conclusion
While NA offers a more comprehensive view of parametric knowledge, IA provides unique insights that NA may miss. Our findings suggest that a synergistic approach, combining both methods, is necessary for a holistic understanding of LMs. Future work should expand these methods to incorporate attention-based neuron attribution, as attention layers play a massive role in knowledge routing that current MLP-focused methods may overlook.
Summary updated