TL;DR
ConceptEvo is a novel interpretability framework that tracks how deep neural networks (DNNs) form and refine internal concepts during the training process, rather than just analyzing them post-training. By projecting different architectures and training epochs into a single, unified semantic space, it allows researchers to diagnose training anomalies—like overfitting or bad hyperparameters—in real-time.
What Problem Does This Paper Address?
Most deep learning interpretability methods are "post-mortem" tools: they analyze a fully trained model to see what it has learned. While useful, this approach leaves a massive blind spot during the training phase. If a model fails to converge, generalizes poorly, or develops biases, engineers often only find out after wasting significant time and compute resources.
To fix this, we need to understand concept evolution—how individual neurons transition from detecting abstract edges to complex, human-understandable concepts (like "dog faces" or "car wheels") across training epochs. However, tracking this is incredibly difficult because different model architectures (e.g., a classic VGG16 vs. a modern ConvNeXt) do not share a common internal language or alignment. This paper introduces a way to bridge that gap and look inside the black box while it's actively learning.
Methods
The authors introduce ConceptEvo, which relies on two primary technical innovations:
- A Unified Semantic Space: To compare different models side-by-side, ConceptEvo creates a shared reference space. It starts with a fully trained "base model" (like VGG19) and embeds its neurons into a space based on how frequently they co-activate when exposed to the same images, using a Word2Vec-style negative sampling approach. To project other models or earlier training epochs into this same space, ConceptEvo uses image embeddings as a bridge. Because all these models are trained on the same dataset, images act as a universal translator to align disparate architectures.
- Quantifying Concept Importance: Not all concept changes matter equally. ConceptEvo uses directional derivatives to calculate how sensitive a final class prediction is to the evolutionary shift of a neuron's concept from epoch $t$ to epoch $t'$. This automatically surfaces the most influential concept transformations.
Datasets
- ILSVRC2012 (ImageNet): The primary dataset used to train and evaluate the vision models. The authors utilized a 10% random sample (120,000 images) to maintain computational efficiency during real-time concept tracking.
- Broden Dataset: Used specifically as a benchmark to extract image segments for baseline comparisons against existing post-hoc interpretability methods like ACE.
Key Insights and Ideas
- Hyperparameter Atrophy: When a model is trained with an excessively high learning rate, ConceptEvo reveals a fascinating failure mode: neurons rapidly lose their concept diversity. Instead of specializing, a massive percentage of neurons (over 30%) "atrophy" and collapse into exclusively detecting generic image backgrounds. ConceptEvo can flag this structural decay long before the training accuracy officially drops to zero.
- The Overfitting Slump: In overfitted models (e.g., a VGG16 stripped of its dropout layers), training accuracy shoots up rapidly to 99%. However, ConceptEvo shows that the underlying concepts evolve at a snail's pace compared to a healthy model. The network is essentially memorizing data configurations rather than refining its conceptual vocabulary.
Key Results
- High Human Alignment: To ensure the unified space actually maps to human logic, the authors ran a large-scale evaluation with 260 Amazon Mechanical Turk participants across 10,950 tasks. Humans consistently identified coherent concept clusters across different models and epochs, yielding a high Area Under the ROC Curve (AUC) ranging from 0.86 to 0.90.
- Causal Validation: The authors tested their importance metric by manually "reverting" the activations of neurons back to their earlier epoch states. Reverting the top 25% "most important" evolving neurons caused a drastic drop in model accuracy, validating that the framework accurately identifies the core concepts driving performance.
- Highly Efficient: ConceptEvo scales linearly with the number of neurons. Projecting a new model epoch into the semantic space takes roughly 1 hour on a single NVIDIA A6000 GPU, making it fast enough to run concurrently alongside standard training loops.
Limitations and Future Work
Currently, ConceptEvo operates under the assumption that an image's representation can be approximated by linear combinations of neuron embeddings. The authors note that more complex, non-linear relationships likely exist within DNN layers, and accounting for these could further improve alignment accuracy.
Additionally, while this work focuses strictly on computer vision backbones (CNNs and ConvNeXt), the authors plan to expand the framework to other domains, including object detectors, reinforcement learning agents, and large language models (LLMs).
Summary updated