Attributing Learned Concepts in Neural Networks to Training Data

Nicholas Konz, Charles Godfrey, Madelyn Shapiro, Jonathan H. Tu, Henry Kvinge, Davis BrownarXiv 2023 · DOI · arXiv

Introduction

Deep learning models are known to develop internal representations of the world, learning concepts such as "fur," "wheels," or "edges" rather than just memorizing pixels. While existing tools like concept probing can locate these representations and data attribution methods like TRAK can identify images influencing specific predictions, a critical question remains: which training examples are responsible for a model learning a specific concept in the first place? This paper introduces a framework for "concept attribution" to trace human-understandable concepts back to their training origins.

Methodology

The researchers developed a pipeline that merges concept probing with high-performance data attribution. First, they train linear probes on the hidden layers of a pre-trained ResNet-18 to identify Concept Activation Vectors (CAVs), which represent the mathematical "direction" of a concept. Second, they apply the TRAK (Tracing with the Randomly-projected After Kernel) attribution method to the probe's predictions, rather than the final model output. To ensure statistical significance, they aggregate results from an ensemble of 20 models.

The concept attribution pipeline: identifying concept directions through probing and then applying dataset attribution to those directions.

Key Insights: The Diffuse Nature of Concepts

The most significant finding is that concept formation is remarkably robust and "diffuse." The researchers identified the 10,000 training images most influential for a specific concept and removed them from the training set. Surprisingly, when the model was retrained, it learned the concept just as effectively. This suggests that the information defining a concept is not concentrated in a few "prototypical" images but is spread across the entire dataset.

Furthermore, the study observed a distinct layer-wise evolution of concepts. Low-level technical concepts, such as spatial frequencies, tend to peak in the middle layers before being integrated into more complex features. In contrast, abstract high-level concepts like "snakes" become increasingly distinct and accurate in deeper layers of the network.

Key Results and Visualizations

The attribution scores provide a visual narrative of how concepts are built. In early layers, images with high attribution for the "snakes" concept often feature scaly textures or winding patterns. In deeper layers, the most influential images transition to full-body photographs of snakes, indicating the network's progression from texture recognition to object-level understanding.

Visualizing influence: the highest and lowest attributed training images for "snakes" and "high-low frequency" concepts across various network layers.

Quantitative analysis confirmed this robustness: removing the top $T$ attributing images (up to $T=10,000$) resulted in no significant drop in the probe's validation accuracy. This redundancy suggests that human-interpretable concepts act as "stable attractors" in the learning process, which the model is incentivized to discover regardless of the specific data subset it encounters.

Limitations and Future Directions

While these findings are compelling, the study was conducted on a ResNet-18 architecture using a 10% subset of ImageNet. Future research is needed to determine if these "diffuse" learning patterns hold true for massive models like Vision Transformers or Large Language Models (LLMs). Additionally, applying this framework to other modalities—such as identifying the training data responsible for a language model's understanding of "sarcasm"—remains an open and promising area of exploration.

Summary updated

Abstract

By now there is substantial evidence that deep learning models learn certain human-interpretable features as part of their internal representations of data. As having the right (or wrong) concepts is critical to trustworthy machine learning systems, it is natural to ask which inputs from the model's original training set were most important for learning a concept at a given layer. To answer this, we combine data attribution methods with methods for probing the concepts learned by a model. Training network and probe ensembles for two concept datasets on a range of network layers, we use the recently developed TRAK method for large-scale data attribution. We find some evidence for convergence, where removing the 10,000 top attributing images for a concept and retraining the model does not change the location of the concept in the network nor the probing sparsity of the concept. This suggests that rather than being highly dependent on a few specific examples, the features that inform the development of a concept are spread in a more diffuse manner across its exemplars, implying robustness in concept formation.
BibTeX
@misc{konz2023attributing,
  author = {Nicholas Konz and Charles Godfrey and Madelyn Shapiro and Jonathan H. Tu and Henry Kvinge and Davis Brown},
  title = {{Attributing Learned Concepts in Neural Networks to Training Data}},
  year = {2023},
  eprint = {2310.03149},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.03149},
}
Figure