ConceptEvo: Interpreting Concept Evolution in Deep Learning Training

Haekyu Park, Seongmin Lee, Benjamin Hoover, Austin P. Wright, Omar Shaikh, Rahul Duggal, Nilaksh Das, Judy Hoffman, Duen Horng Chau

Introduction

Deep learning models are often criticized as "black boxes," but the most mysterious part isn't just how they work once finished—it’s how they learn in the first place. While we have many tools to interpret a fully trained model, we have surprisingly few that show us the "inception" and evolution of concepts during the training process itself. In their CIKM ’23 paper, researchers from Georgia Tech introduce ConceptEvo, a framework designed to track how neurons move from detecting simple edges to complex, human-understandable concepts like "dog faces" or "car wheels" as training progresses.

The Challenge of Training Dynamics

Most interpretability research is "post-hoc," analyzing a model after it has finished training. While useful for debugging a final product, this approach offers little insight into training dynamics. If a model fails to converge or generalizes poorly, post-hoc analysis cannot pinpoint when or why the learning process failed. Furthermore, comparing different models (e.g., VGG19 vs. ConvNeXt) is difficult because their internal representations are unaligned. ConceptEvo addresses these gaps by visualizing and quantifying the "concept trajectory" of neurons across different epochs and architectures.

The ConceptEvo Framework

The authors propose a two-step approach to create a Unified Semantic Space:

  1. Building the Base Space: Using a fully trained "base model" (like VGG19), they identify concepts each neuron detects by finding the stimuli that activate them most strongly. A Word2Vec-inspired embedding technique maps these neurons into a 2D space where similar concepts are clustered.
  2. Aligning Other Models: To compare a new model or an earlier epoch, ConceptEvo uses image embeddings as a bridge. If neurons in different models respond to the same images, they are projected to the same location in the unified space.
ConceptEvo maps neurons from different architectures into a unified semantic space, identifying shared concepts like "dog face" or "car wheel."

To determine if a concept's evolution matters, the authors use an algorithm based on directional derivatives. It measures how sensitive a class prediction is to the change in a neuron's activation as it evolves from an abstract shape to a specific object.

Key Insights: Context and Atrophy

ConceptEvo reveals surprising details about how models learn. For the "bow tie" class, the framework discovered that the most important evolving concepts were human necks and faces rather than the ties themselves. This highlights how models rely on contextual "human" concepts to identify accessories.

The tool also helps diagnose training failures. The researchers found that excessively high learning rates cause "concept atrophy," where neurons that should learn diverse features instead collapse into detecting only low-level background textures.

Visualization of concept atrophy during training where neurons collapse into a single point, losing predictive power.

Results and Performance

In a study with 260 participants, humans agreed with ConceptEvo’s concept groupings with high accuracy (AUC scores between 0.86 and 0.98). Furthermore, "reverting" the evolution of high-importance neurons caused a significantly larger drop in model accuracy than reverting random neurons, validating the framework's importance metrics. ConceptEvo is also efficient; projecting a new model onto the semantic space takes about an hour, making it a viable tool for real-time training monitoring.

Conclusion

By turning the "black box" of training into a "glass box," ConceptEvo provides a roadmap for researchers to not only build better models but to understand the very nature of machine learning. Future work aims to expand this framework beyond CNNs to Transformers and non-visual domains.

Summary updated

Abstract

Deep neural networks (DNNs) have been widely used for decision making, prompting a surge of interest in interpreting how these complex models work. Recent literature on DNN interpretation has revolved around already-trained models; however, much less research focuses on interpreting how the models evolve as they are trained. Interpreting model evolution is crucial to monitor network training and can aid proactive decisions about necessary interventions. In this work, we present ConceptEvo, a general interpretation framework for DNNs that reveals the inception and evolution of detected concepts during training. Through a large-scale human evaluation with 260 participants and quantitative experiments, we show that ConceptEvo discovers evolution across different models that are meaningful to humans, helpful for early-training intervention decisions, and crucial to the prediction for a given class.
BibTeX
@inproceedings{park2023conceptevo,
  author = {Haekyu Park and Seongmin Lee and Benjamin Hoover and Austin P. Wright and Omar Shaikh and Rahul Duggal and Nilaksh Das and Judy Hoffman and Duen Horng Chau},
  title = {{ConceptEvo: Interpreting Concept Evolution in Deep Learning Training}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2023},
  doi = {10.1145/3583780.3614819},
}
Figure