Concept Evolution in Deep Learning Training: A Unified Interpretation Framework and Discoveries

Haekyu Park, Seongmin Lee, Benjamin Hoover, Austin P. Wright, Omar Shaikh, Rahul Duggal, Nilaksh Das, Kevin Li, Judy Hoffman, Duen Horng Chau

TL;DR

This paper introduces CONCEPTEVO, a novel framework for understanding how concepts emerge and evolve within deep neural networks during training. By creating a unified semantic space to compare models and identifying key concept changes, CONCEPTEVO offers crucial insights into model behavior, potential training issues, and concept importance for predictions.

What Problem Does This Paper Address?

Interpreting deep neural networks (DNNs) is vital for building trust, debugging, and ensuring fairness. While many methods focus on understanding trained models, there's a significant gap in understanding the dynamic process of concept learning during training. Existing approaches often analyze models only after training, missing critical insights into how concepts form, change, and potentially lead to issues like poor generalization or convergence failures. This lack of insight into the training trajectory can lead to wasted resources if problems are only discovered late in the process.

Methods

CONCEPTEVO tackles this by introducing a two-step framework:

  1. Unified Semantic Space Creation:
    • Base Semantic Space: A well-trained model is used as a reference to define a semantic space. Neurons are embedded based on their activation patterns to similar concepts. This involves identifying "stimuli" (images that strongly activate a neuron) and then learning embeddings for neurons based on how frequently they are co-activated by the same images. This uses a negative sampling approach similar to Word2Vec.
  2. Projection: Concepts from other models (at various training stages) are then projected onto this base semantic space. This allows for direct, side-by-side comparison of concept evolution across different models and training epochs. Image embeddings act as a bridge, approximating neuron embeddings.
  3. Concept Evolution Discovery and Quantification:
    • Concept Evolution Importance: The framework identifies how concept evolution impacts class predictions. It quantifies this by measuring the sensitivity of a class prediction to a neuron's concept change. A significant drop in prediction accuracy when a concept evolution is "reverted" indicates its importance.
  4. Discoveries: This allows for identifying training problems, such as concept diversity degradation due to incompatible hyperparameters (e.g., high learning rates) or slow concept evolution in overfitted models.

Datasets

  • ImageNet (ILSVRC2012): Used for training and evaluating the models.
  • Broden dataset: Used for concept-based explanations in the comparison with existing methods.

Key Insights and Ideas

  • Training Dynamics Matter: Understanding how concepts evolve during training is as crucial as understanding the final learned concepts.
  • Unified View: The ability to compare concept evolution across different models and training stages in a single semantic space is a significant advancement.
  • Hyperparameter Impact: CONCEPTEVO can directly reveal how hyperparameters like learning rates can negatively impact concept diversity, leading to model deficiencies even before accuracy drops significantly.
  • Overfitting Slowdown: Overfitted models show slower concept evolution despite rapid increases in training accuracy, suggesting a disconnect between memorization and true concept learning.
  • Concept Importance for Prediction: The framework provides a quantitative measure of how critical specific concept evolutions are for a model's ability to predict a given class.

Key Results

  • Human Evaluation: A large-scale human evaluation with 260 participants demonstrated that CONCEPTEVO's identified concepts are semantically meaningful and align well across different models and training epochs (AUC scores ranging from 0.86 to 0.90).
  • Concept Evolution Importance: Reverting higher-importance concept evolutions led to larger drops in top-1 training accuracy, validating CONCEPTEVO's ability to quantify concept importance. For instance, reverting the most important evolutions (0-25th percentile) caused a significant accuracy drop, while reverting random evolutions had a less pronounced effect.
  • Discovery of Training Issues:
    • Concept Diversity Loss: CONCEPTEVO identified that overly large learning rates in VGG16 and ConvNeXt models led to a drastic reduction in concept diversity, with many neurons predominantly detecting "background" concepts.
  • Slow Evolution in Overfitting: Overfitted models (VGG16 and ConvNeXt without dropout) showed significantly slower movement of neuron embeddings compared to well-trained models, indicating sluggish concept evolution.
  • Comparison with Existing Methods: CONCEPTEVO effectively aligns concepts across training epochs, while methods like NeuroCartography and ACE showed misalignment (flipping, rotation, shifting) of concepts over time.

Limitations and Future Work

  • Limitations:
    • The current framework assumes linear combinations of neurons can represent images.
  • The computational cost of Step 2.1 (image embedding) can be significant (up to 3 hours).
  • Future Work:
    • Expand to other model types like object detectors, reinforcement learning systems, and language models.
  • Enhance concept alignment across different models by exploring non-linear relationships beyond linear combinations.
  • Develop more sophisticated methods to handle complex relationships between images and neurons.

Summary updated

Abstract

We present ConceptEvo, a unified interpretation framework for deep neural networks (DNNs) that reveals the inception and evolution of learned concepts during training. Our work addresses a critical gap in DNN interpretation research, as existing methods primarily focus on post-training interpretation. ConceptEvo introduces two novel technical contributions: (1) an algorithm that generates a unified semantic space, enabling side-by-side comparison of different models during training, and (2) an algorithm that discovers and quantifies important concept evolutions for class predictions. Through a large-scale human evaluation and quantitative experiments, we demonstrate that ConceptEvo successfully identifies concept evolutions across different models, which are not only comprehensible to humans but also crucial for class predictions. ConceptEvo is applicable to both modern DNN architectures, such as ConvNeXt, and classic DNNs, such as VGGs and InceptionV3.
BibTeX
@inproceedings{park2023concept,
  author = {Haekyu Park and Seongmin Lee and Benjamin Hoover and Austin P. Wright and Omar Shaikh and Rahul Duggal and Nilaksh Das and Kevin Li and Judy Hoffman and Duen Horng Chau},
  title = {{Concept Evolution in Deep Learning Training: A Unified Interpretation Framework and Discoveries}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2023},
  doi = {10.1145/3583780.3614819},
}
Figure