Introduction
Deep learning models are often criticized as "black boxes," but the most mysterious part isn't just how they work once finished—it’s how they learn in the first place. While we have many tools to interpret a fully trained model, we have surprisingly few that show us the "inception" and evolution of concepts during the training process itself. In their CIKM ’23 paper, researchers from Georgia Tech introduce ConceptEvo, a framework designed to track how neurons move from detecting simple edges to complex, human-understandable concepts like "dog faces" or "car wheels" as training progresses.
The Challenge of Training Dynamics
Most interpretability research is "post-hoc," analyzing a model after it has finished training. While useful for debugging a final product, this approach offers little insight into training dynamics. If a model fails to converge or generalizes poorly, post-hoc analysis cannot pinpoint when or why the learning process failed. Furthermore, comparing different models (e.g., VGG19 vs. ConvNeXt) is difficult because their internal representations are unaligned. ConceptEvo addresses these gaps by visualizing and quantifying the "concept trajectory" of neurons across different epochs and architectures.
The ConceptEvo Framework
The authors propose a two-step approach to create a Unified Semantic Space:
- Building the Base Space: Using a fully trained "base model" (like VGG19), they identify concepts each neuron detects by finding the stimuli that activate them most strongly. A Word2Vec-inspired embedding technique maps these neurons into a 2D space where similar concepts are clustered.
- Aligning Other Models: To compare a new model or an earlier epoch, ConceptEvo uses image embeddings as a bridge. If neurons in different models respond to the same images, they are projected to the same location in the unified space.
To determine if a concept's evolution matters, the authors use an algorithm based on directional derivatives. It measures how sensitive a class prediction is to the change in a neuron's activation as it evolves from an abstract shape to a specific object.
Key Insights: Context and Atrophy
ConceptEvo reveals surprising details about how models learn. For the "bow tie" class, the framework discovered that the most important evolving concepts were human necks and faces rather than the ties themselves. This highlights how models rely on contextual "human" concepts to identify accessories.
The tool also helps diagnose training failures. The researchers found that excessively high learning rates cause "concept atrophy," where neurons that should learn diverse features instead collapse into detecting only low-level background textures.
Results and Performance
In a study with 260 participants, humans agreed with ConceptEvo’s concept groupings with high accuracy (AUC scores between 0.86 and 0.98). Furthermore, "reverting" the evolution of high-importance neurons caused a significantly larger drop in model accuracy than reverting random neurons, validating the framework's importance metrics. ConceptEvo is also efficient; projecting a new model onto the semantic space takes about an hour, making it a viable tool for real-time training monitoring.
Conclusion
By turning the "black box" of training into a "glass box," ConceptEvo provides a roadmap for researchers to not only build better models but to understand the very nature of machine learning. Future work aims to expand this framework beyond CNNs to Transformers and non-visual domains.
Summary updated