Introduction
As deep neural networks (DNNs) are increasingly deployed in high-stakes domains such as medicine and autonomous driving, the need for interpretable models becomes paramount. Concept-based explanations have emerged as a powerful tool to bridge the gap between abstract latent representations and human-understandable terms. By mapping internal model states to concepts like "stripes" or "curvature," researchers can gain insights into the decision-making process of complex black-box models.
The Linearity Trap
The current state-of-the-art in concept-based interpretability, Concept Activation Vectors (CAVs), relies on the fundamental assumption of linear separability. CAVs assume that all instances of a concept are grouped in a way that can be bisected by a single hyperplane in the model's latent space.
However, this assumption often fails in practice. A model may represent a single concept across multiple disjoint "islands" or clusters. For instance, the concept of "stripes" might be encoded differently for a zebra than for a tiger, leading to a non-linear distribution in the representation space. Forcing a linear boundary through these scattered clusters results in inaccurate explanations and fails to capture the true complexity of the model's internal logic.
Methodology: Concept Activation Regions (CARs)
To address the limitations of linear methods, the authors introduce Concept Activation Regions (CARs). This framework generalizes concept-based explanations by allowing for non-linear boundaries in the latent space using kernel-based classifiers.
The CAR framework consists of several key components:
- Kernel-Based Classifiers: Instead of linear hyperplanes, CARs utilize Support Vector Classifiers (SVCs) with non-linear kernels. This enables the definition of complex "regions" of concept presence.
- Concept Density: A mathematical density function is defined to determine the likelihood of a concept's presence for a given input. High-density areas indicate a strong influence of the concept on the model's prediction.
- TCAR Scores: The Testing Concept Activation Region (TCAR) score provides a global metric to quantify the overlap between a specific class and a concept region.
- Isometry Invariance: CARs are theoretically robust; using radial kernels ensures that explanations remain consistent even under rotations or shifts of the latent space, provided the underlying geometry is preserved.
Experimental Validation
The efficacy of CARs was validated across four diverse domains:
- MNIST: Geometric concepts like "Loop" and "Vertical Line" for handwritten digits.
- MIT-BIH ECG: Expert-defined medical concepts for heartbeat recordings.
- CUB (Birds): Visual attribute concepts (e.g., wing color) for bird species classification.
- SEER (Prostate Cancer): Investigating whether a model could autonomously rediscover clinical grading systems for cancer severity.
Key Results and Insights
- Superior Accuracy: CAR classifiers significantly outperformed linear CAVs, particularly in deeper network layers where representations are most abstract and non-linear.
- Better Human Alignment: TCAR scores showed near-perfect correlation ($r \approx 1.00$) with ground-truth human annotations in MNIST and ECG tasks, whereas the older TCAV method was much noisier ($r \approx 0.60$).
- Autonomous Scientific Discovery: In the prostate cancer study, CARs revealed that the model had implicitly organized its latent space according to the official medical grading system, despite never being shown those labels during training.
- Meaningful Feature Maps: CAR-based saliency maps successfully highlighted specific parts of an image corresponding to a concept, such as the "loop" of an '8'.
Conclusion and Future Work
Concept Activation Regions represent a significant step toward more faithful and "human-compatible" AI explanations. By moving away from oversimplified linear models toward a framework that respects the true complexity of deep learning, CARs provide more scientifically grounded insights. Future work will address computational scaling for very large datasets and explore making early network layers more interpretable through concept-based contrastive learning.
Summary updated