Abstract
Explainable AI (XAI) methods like Testing with Concept Activation Vectors (TCAV) are widely used to interpret neural network decisions. However, the stochastic nature of selecting random reference data leads to variability in results. This paper provides a rigorous theoretical and empirical analysis of this variability. We find that while the variance of the Concept Activation Vectors (CAVs) decreases with more random data ($1/N$), the resulting TCAV scores can remain unstable due to "borderline" data points. Our results suggest that averaging multiple small runs is more effective for stabilizing explanations than increasing the sample size of a single run.
Introduction
As deep learning models become increasingly complex, the need for transparency grows. Testing with Concept Activation Vectors (TCAV) is a popular technique that quantifies the influence of high-level concepts (e.g., "stripes") on a model's prediction (e.g., "zebra"). By finding a "direction" in the network's latent space that represents a concept, TCAV provides human-interpretable explanations.
A significant hurdle for TCAV is its sensitivity to the choice of random "negative" images used to define the concept's boundary. This randomness introduces instability, where different runs of the same algorithm yield different importance scores. This paper investigates the trade-offs in resource allocation: is it better to use a large set of random samples in one run, or to average results across multiple smaller runs?
The TCAV Framework
The TCAV process involves three main steps: extracting feature vectors from a specific layer, training a linear classifier to separate concept examples from random examples, and calculating the sensitivity of the model's output to changes in the concept direction.
Methodology
The authors analyze CAV variability through two lenses:
- Theoretical Analysis: By modeling CAV construction as a logistic regression problem in an "infinitely imbalanced regime," they prove the asymptotic normality of the CAV estimator. They demonstrate that the vector's variance scales as $O(1/N)$, where $N$ is the number of random samples.
- Empirical Validation: The study compares three common ways to compute CAVs: Logistic Regression, Linear SVMs, and the "Difference of Means" (a simpler geometric approach). These are tested across image (ImageNet/Broden), tabular (UCI Adult), and text (IMDB Sentiment) datasets to ensure generalizability.
Key Insights: The TCAV Score Paradox
The research reveals a counter-intuitive finding: even as the CAV direction becomes perfectly stable with more data, the final TCAV score (the percentage of images influenced by the concept) can remain volatile.
This paradox is driven by "borderline points"—data samples positioned near the decision boundary in the feature space. Even infinitesimal shifts in the CAV direction can cause these points to flip their classification, maintaining the variance of the final score at a constant $O(1)$ level regardless of the sample size $N$.
Practical Results and Guidelines
The paper offers clear strategies for practitioners based on their specific goals:
- To stabilize TCAV scores: It is more compute-efficient to run several independent simulations with a modest number of random samples and average the resulting scores, rather than performing one massive run.
- To stabilize CAV directions: If the goal is model steering or bias mitigation (where the vector itself is the primary output), increasing the number of samples ($N$) per run is the superior approach.
Conclusion
This work establishes the first formal bounds on the stability of concept-based explanations. While increasing random data helps stabilize the underlying vectors, the discrete nature of the TCAV score requires an ensemble-based approach for reliability. Future work remains to explore how different optimization algorithms, such as SGD, contribute to this variability and to further investigate the high empirical stability of simpler methods like the "Difference of Means."
Summary updated