Introduction
Deep neural networks are often criticized as "black boxes" because their internal decision-making processes are difficult for humans to interpret. While researchers have developed Concept Activation Vectors (CAVs) to identify directions in a model's feature space that correspond to specific concepts (e.g., "stripes" or "blue"), a significant bottleneck remains: identifying what these vectors actually represent. Traditionally, this requires manual inspection of the images that most strongly activate a CAV, a process that is subjective, time-consuming, and prone to error.
This paper introduces an automated pipeline designed to translate these mathematical directions into human-readable text. By leveraging joint text-image embeddings and focusing on specific receptive fields rather than full images, the authors provide a scalable method for interpreting the hidden concepts within neural networks.
The Problem: Concept Noise and Manual Interpretation
The primary challenge in concept description is "concept noise." When a model's early layer detects a simple feature, such as a blue texture, and the researcher looks at the full images that triggered that detection (e.g., sharks or whales), they might incorrectly label the concept as "marine life." In reality, the neuron was only responding to the color blue. This discrepancy between the global image content and the local feature activation makes automated labeling difficult.
Methodology: The Three-Step Pipeline
The authors propose a robust framework to automate the description of discovered concepts:
1. Concept Discovery
The process begins by using unsupervised methods to identify a set of CAVs within a pre-trained model, such as a ResNet50. These vectors represent the internal features the model has learned to prioritize.
2. Visual Description and the Receptive Field Innovation
To solve the problem of concept noise, the authors introduce a strategy focused on receptive fields—the specific patches of an image that a network "sees" at a given layer. Instead of analyzing the entire image, they employ a strategy called $F_{mean \to max}$. This method selects images with high overall activation for a CAV but then crops them to the specific patch with the highest concept score. This ensures that the subsequent text generation is based only on the visual data that actually triggered the model's internal response.
3. Textual Description via CLIP and SoftWPMI
The cropped patches are mapped into a joint text-image latent space using CLIP. The system then compares these visual embeddings against a large candidate vocabulary (e.g., the "google20k" dataset). To determine the most accurate label, the authors use Soft Weighted Pointwise Mutual Information (SoftWPMI). This metric ranks words based on how well they describe the mutual information across the entire set of representative image patches. Finally, a "common description" is derived by computing a weighted average of the top-ranked words, providing a single, concise label rather than a list of synonyms.
Experimental Results and Insights
The authors validated their approach across several benchmarks, including MNIST, CIFAR10, and Animals with Attributes 2 (AwA2).
- The Power of the Patch: The use of receptive fields proved crucial. In early layers, the system successfully distinguished between simple color concepts (like "blue") and complex environmental concepts (like "underwater") by focusing on the specific textures being processed.
- Label Accuracy: On CIFAR10, the pipeline successfully matched CAVs to their ground-truth class labels (e.g., identifying the "airplane" CAV as "aircraft"). On the more complex AwA2 dataset, the system achieved 87% accuracy in concept discovery, providing meaningful labels like "green" for cows (referencing the grass in the background).
- Detecting Model Bias: The method was tested on a "Dark Cats vs. Dogs" dataset to see if it could uncover simplicity bias. The system correctly identified that the model was using "darkness" as a proxy for the "cat" class, revealing a flaw in the training data that would have been difficult to detect through standard performance metrics.
Limitations and Future Work
While the pipeline is highly effective, it is constrained by the capabilities of the underlying joint embedding. If CLIP has not learned a specific niche concept, the system cannot describe it. Additionally, the current implementation relies on a predefined vocabulary and is limited to single-word labels.
Future research directions include moving beyond single words toward more complex natural language descriptions and exploring more fine-grained input selection to further refine the accuracy of the generated labels. By automating the interpretation of CAVs, this work represents a significant step toward making deep learning models more transparent and accountable.
Summary updated