Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods

Haeun Yu, Pepa Atanasova, Isabelle Augenstein

Abstract

Large Language Models (LLMs) store vast amounts of knowledge within their parameters, yet the mechanisms of this storage remain opaque. This paper introduces a unified framework to bridge the gap between Instance Attribution (IA)—identifying influential training examples—and Neuron Attribution (NA)—pinpointing specific active internal units. By developing alignment methods like NA-Instances and IA-Neurons, we evaluate the faithfulness and consistency of these explanations across diverse NLP tasks. Our findings reveal a significant disconnect between the two attribution lenses, suggesting that they capture distinct facets of a model's parametric knowledge and that a hybrid approach is necessary for true transparency.

Introduction

As LLMs scale, understanding the "black box" of their internal weights becomes critical. Current research typically employs either Instance Attribution (IA), which identifies specific training examples that influenced a prediction, or Neuron Attribution (NA), which identifies specific neurons (usually in MLP layers) that activate for a specific fact. However, these two fields have largely operated in silos. This study investigates whether the "influential instances" found by IA actually activate the "knowledge neurons" found by NA. Without a unified way to compare them, it is difficult to determine if our explanations are faithful to the model's actual internal mechanics.

Methods

The researchers developed a framework to make IA and NA comparable through two primary alignment techniques:

  1. NA-Instances: A novel method that uses Neuron Attribution to find training instances. It identifies training examples sharing the most similar neuron activation patterns with a test instance using a new metric called Discounted Cumulative Neuron Similarity (DCNS).
  2. IA-Neurons: Conversely, this method takes the most influential instances found by IA and identifies which of their neurons are most active.
Overview of the unified framework for comparing Instance and Neuron Attribution, including alignment methods, faithfulness tests, and fine-tuning procedures.

Faithfulness was assessed via Sufficiency and Comprehensiveness tests on neuron activations, alongside fine-tuning experiments using influential training instances. The framework was evaluated on three diverse datasets—MNLI, AVeriTeC, and CoS-QA—using OPT-125m, Pythia-410m, and BLOOM-560m models.

Key Results and Insights

A primary finding of this research is the "Great Disconnect": there is remarkably low overlap (often under 20%) between the training instances identified by IA and those identified by NA-Instances. This suggests the two methods are likely capturing different facets of the model's "memory."

Proportion of important neurons identified by NA and IF-Neurons across different datasets, highlighting the low overlap between the two methods.

Additional key insights include:

  • NA is More Comprehensive: Neuron Attribution generally revealed more diverse and comprehensive information about the model's internal state than Instance Attribution. IA methods (like Influence Functions) tend to converge on a very small, homogenous set of influential instances, whereas NA-Instances identified a much broader set of unique examples.
  • Diversity in Fine-tuning: Interestingly, fine-tuning on the least similar instances (according to NA) sometimes yielded better performance than the most similar ones, likely because the "least" group was more diverse.
  • Artifact Detection: NA-Instances proved significantly better at identifying "dataset artifacts" (shortcuts or biases learned during training) compared to traditional IA methods.
  • The Role of Attention: Even when suppressing almost all neurons in the MLP layers, models often still predicted correctly, suggesting a significant portion of knowledge is stored in attention heads rather than just MLP "knowledge neurons."

Discussion and Conclusion

The study highlights that while NA is currently a more powerful tool for revealing internal mechanisms, IA provides human-interpretable context that NA lacks. The results suggest that focusing solely on MLP layers may be insufficient, and future work must incorporate attention-based attribution to get a complete picture of parametric knowledge. Ultimately, the paper concludes that a synergistic approach combining both IA and NA is the most promising path forward for achieving transparent and interpretable AI.

Summary updated

Abstract

Language Models (LMs) acquire parametric knowledge from their training process, embedding it within their weights. The increasing scalability of LMs, however, poses significant challenges for understanding a model's inner workings and further for updating or correcting this embedded knowledge without the significant cost of retraining. This underscores the importance of unveiling exactly what knowledge is stored and its association with specific model components. Instance Attribution (IA) and Neuron Attribution (NA) offer insights into this training-acquired knowledge, though they have not been compared systematically. Our study introduces a novel evaluation framework to quantify and compare the knowledge revealed by IA and NA. To align the results of the methods we introduce the attribution method NA-Instances to apply NA for retrieving influential training instances, and IA-Neurons to discover important neurons of influential instances discovered by IA. We further propose a comprehensive list of faithfulness tests to evaluate the comprehensiveness and sufficiency of the explanations provided by both methods. Through extensive experiments and analysis, we demonstrate that NA generally reveals more diverse and comprehensive information regarding the LM's parametric knowledge compared to IA. Nevertheless, IA provides unique and valuable insights into the LM's parametric knowledge, which are not revealed by NA. Our findings further suggest the potential of a synergistic approach of combining the diverse findings of IA and NA for a more holistic understanding of an LM's parametric knowledge.
BibTeX
@inproceedings{yu2024revealing,
  author = {Haeun Yu and Pepa Atanasova and Isabelle Augenstein},
  title = {{Revealing the Parametric Knowledge of Language Models: A Unified Framework for Attribution Methods}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.444},
}
Figure