Concept Evolution in Deep Learning Training: A Unified Interpretation Framework and Discoveries

Haekyu Park, Seongmin Lee, Benjamin Hoover, Austin P. Wright, Omar Shaikh, Rahul Duggal, Nilaksh Das, Kevin Li, Judy Hoffman, Duen Horng Chau

TL;DR

ConceptEvo is a novel interpretability framework that tracks how deep neural networks (DNNs) form and refine internal concepts during the training process, rather than just analyzing them post-training. By projecting different architectures and training epochs into a single, unified semantic space, it allows researchers to diagnose training anomalies—like overfitting or bad hyperparameters—in real-time.

What Problem Does This Paper Address?

Most deep learning interpretability methods are "post-mortem" tools: they analyze a fully trained model to see what it has learned. While useful, this approach leaves a massive blind spot during the training phase. If a model fails to converge, generalizes poorly, or develops biases, engineers often only find out after wasting significant time and compute resources.

To fix this, we need to understand concept evolution—how individual neurons transition from detecting abstract edges to complex, human-understandable concepts (like "dog faces" or "car wheels") across training epochs. However, tracking this is incredibly difficult because different model architectures (e.g., a classic VGG16 vs. a modern ConvNeXt) do not share a common internal language or alignment. This paper introduces a way to bridge that gap and look inside the black box while it's actively learning.

Methods

The authors introduce ConceptEvo, which relies on two primary technical innovations:

  1. A Unified Semantic Space: To compare different models side-by-side, ConceptEvo creates a shared reference space. It starts with a fully trained "base model" (like VGG19) and embeds its neurons into a space based on how frequently they co-activate when exposed to the same images, using a Word2Vec-style negative sampling approach. To project other models or earlier training epochs into this same space, ConceptEvo uses image embeddings as a bridge. Because all these models are trained on the same dataset, images act as a universal translator to align disparate architectures.
  2. Quantifying Concept Importance: Not all concept changes matter equally. ConceptEvo uses directional derivatives to calculate how sensitive a final class prediction is to the evolutionary shift of a neuron's concept from epoch $t$ to epoch $t'$. This automatically surfaces the most influential concept transformations.

Datasets

  • ILSVRC2012 (ImageNet): The primary dataset used to train and evaluate the vision models. The authors utilized a 10% random sample (120,000 images) to maintain computational efficiency during real-time concept tracking.
  • Broden Dataset: Used specifically as a benchmark to extract image segments for baseline comparisons against existing post-hoc interpretability methods like ACE.

Key Insights and Ideas

  • Hyperparameter Atrophy: When a model is trained with an excessively high learning rate, ConceptEvo reveals a fascinating failure mode: neurons rapidly lose their concept diversity. Instead of specializing, a massive percentage of neurons (over 30%) "atrophy" and collapse into exclusively detecting generic image backgrounds. ConceptEvo can flag this structural decay long before the training accuracy officially drops to zero.
  • The Overfitting Slump: In overfitted models (e.g., a VGG16 stripped of its dropout layers), training accuracy shoots up rapidly to 99%. However, ConceptEvo shows that the underlying concepts evolve at a snail's pace compared to a healthy model. The network is essentially memorizing data configurations rather than refining its conceptual vocabulary.
ConceptEvo tracks healthy concept formation versus hyperparameter failure and overfitting paths during training.

Key Results

  • High Human Alignment: To ensure the unified space actually maps to human logic, the authors ran a large-scale evaluation with 260 Amazon Mechanical Turk participants across 10,950 tasks. Humans consistently identified coherent concept clusters across different models and epochs, yielding a high Area Under the ROC Curve (AUC) ranging from 0.86 to 0.90.
  • Causal Validation: The authors tested their importance metric by manually "reverting" the activations of neurons back to their earlier epoch states. Reverting the top 25% "most important" evolving neurons caused a drastic drop in model accuracy, validating that the framework accurately identifies the core concepts driving performance.
  • Highly Efficient: ConceptEvo scales linearly with the number of neurons. Projecting a new model epoch into the semantic space takes roughly 1 hour on a single NVIDIA A6000 GPU, making it fast enough to run concurrently alongside standard training loops.
Figure 1: ConceptEvo creates a unified semantic space that enables side-by-side comparison of different models dur- ing training (top: VGG19; middle: InceptionV3; bottom: Con- vNeXt). ConceptEvo embeds and aligns neurons (dots) that detect similar concepts (e.g., dog face, circle, car wheel) to similar locations.

Limitations and Future Work

Currently, ConceptEvo operates under the assumption that an image's representation can be approximated by linear combinations of neuron embeddings. The authors note that more complex, non-linear relationships likely exist within DNN layers, and accounting for these could further improve alignment accuracy.

Additionally, while this work focuses strictly on computer vision backbones (CNNs and ConvNeXt), the authors plan to expand the framework to other domains, including object detectors, reinforcement learning agents, and large language models (LLMs).

Summary updated

Abstract

We present ConceptEvo, a unified interpretation framework for deep neural networks (DNNs) that reveals the inception and evolution of learned concepts during training. Our work addresses a critical gap in DNN interpretation research, as existing methods primarily focus on post-training interpretation. ConceptEvo introduces two novel technical contributions: (1) an algorithm that generates a unified semantic space, enabling side-by-side comparison of different models during training, and (2) an algorithm that discovers and quantifies important concept evolutions for class predictions. Through a large-scale human evaluation and quantitative experiments, we demonstrate that ConceptEvo successfully identifies concept evolutions across different models, which are not only comprehensible to humans but also crucial for class predictions. ConceptEvo is applicable to both modern DNN architectures, such as ConvNeXt, and classic DNNs, such as VGGs and InceptionV3.
BibTeX
@inproceedings{park2023concept,
  author = {Haekyu Park and Seongmin Lee and Benjamin Hoover and Austin P. Wright and Omar Shaikh and Rahul Duggal and Nilaksh Das and Kevin Li and Judy Hoffman and Duen Horng Chau},
  title = {{Concept Evolution in Deep Learning Training: A Unified Interpretation Framework and Discoveries}},
  booktitle = {ACM International Conference on Information and Knowledge Management},
  year = {2023},
  doi = {10.1145/3583780.3614819},
}
Figure