Introduction
Deep learning models are known to develop internal representations of the world, learning concepts such as "fur," "wheels," or "edges" rather than just memorizing pixels. While existing tools like concept probing can locate these representations and data attribution methods like TRAK can identify images influencing specific predictions, a critical question remains: which training examples are responsible for a model learning a specific concept in the first place? This paper introduces a framework for "concept attribution" to trace human-understandable concepts back to their training origins.
Methodology
The researchers developed a pipeline that merges concept probing with high-performance data attribution. First, they train linear probes on the hidden layers of a pre-trained ResNet-18 to identify Concept Activation Vectors (CAVs), which represent the mathematical "direction" of a concept. Second, they apply the TRAK (Tracing with the Randomly-projected After Kernel) attribution method to the probe's predictions, rather than the final model output. To ensure statistical significance, they aggregate results from an ensemble of 20 models.
Key Insights: The Diffuse Nature of Concepts
The most significant finding is that concept formation is remarkably robust and "diffuse." The researchers identified the 10,000 training images most influential for a specific concept and removed them from the training set. Surprisingly, when the model was retrained, it learned the concept just as effectively. This suggests that the information defining a concept is not concentrated in a few "prototypical" images but is spread across the entire dataset.
Furthermore, the study observed a distinct layer-wise evolution of concepts. Low-level technical concepts, such as spatial frequencies, tend to peak in the middle layers before being integrated into more complex features. In contrast, abstract high-level concepts like "snakes" become increasingly distinct and accurate in deeper layers of the network.
Key Results and Visualizations
The attribution scores provide a visual narrative of how concepts are built. In early layers, images with high attribution for the "snakes" concept often feature scaly textures or winding patterns. In deeper layers, the most influential images transition to full-body photographs of snakes, indicating the network's progression from texture recognition to object-level understanding.
Quantitative analysis confirmed this robustness: removing the top $T$ attributing images (up to $T=10,000$) resulted in no significant drop in the probe's validation accuracy. This redundancy suggests that human-interpretable concepts act as "stable attractors" in the learning process, which the model is incentivized to discover regardless of the specific data subset it encounters.
Limitations and Future Directions
While these findings are compelling, the study was conducted on a ResNet-18 architecture using a 10% subset of ImageNet. Future research is needed to determine if these "diffuse" learning patterns hold true for massive models like Vision Transformers or Large Language Models (LLMs). Additionally, applying this framework to other modalities—such as identifying the training data responsible for a language model's understanding of "sarcasm"—remains an open and promising area of exploration.
Summary updated