Explainable AI (XAI) often focuses on highlighting important pixels, but humans reason in concepts. A doctor diagnosing an X-ray doesn't look for "Pixel (242, 11)"; they look for a "nodule" or "calcification." While recent methods like Concept Bottleneck Models and Concept Activation Vectors (CAVs) attempt to bridge this gap, they frequently lack the statistical rigor needed for high-stakes domains.
This paper introduces a rigorous statistical framework for understanding why "black-box" machine learning models make specific decisions. By focusing on semantic concepts—human-understandable ideas like "furry," "stripes," or "wings"—the authors provide a way to rank concept importance with formal statistical guarantees.
TL;DR
The authors propose measuring the importance of semantic concepts in predictive models using a "Testing by Betting" approach. By framing importance as a conditional independence test and using sequential e-values, they can rank how much a concept (e.g., "whiskers") influences a model's prediction (e.g., "cat"), providing rigorous control over false positives.
The Problem with Current Explanations
Existing concept-based methods usually cannot quantify the probability that a concept is wrongly identified as important (the False Positive Rate). Furthermore, many approaches require training a simpler "surrogate" model to explain the complex one, which might not accurately reflect how the original black box operates. This paper aims to provide statistical guarantees—like Type I error and False Discovery Rate control—directly on the original, non-linear model without retraining.
Methods: Testing by Betting
The core innovation treats semantic importance as a Conditional Independence problem. If a model’s prediction ($Y$) is independent of a concept ($Z_j$) given all other concepts ($Z_{-j}$), then that concept isn't contributing unique information to the decision.
To test this, the authors utilize Testing by Betting:
- The Game: Imagine a "bettor" starting with $1. At each step, they observe a new data point and place a wager against the null hypothesis (which states the concept is unimportant).
- Wealth Process: If the concept is truly important, the bettor’s wealth grows. If it reaches a threshold (e.g., $1/\alpha$, where $\alpha$ is the significance level), the null hypothesis is rejected, and the concept is deemed "important."
- Sequential Testing: Unlike traditional p-values that require a fixed dataset size, this betting approach is sequential and stops as soon as it gathers enough evidence. This naturally creates a rank of importance: concepts that make the bettor "rich" the fastest are the most critical.
The authors introduce two specific algorithms:
- C-SKIT (Conditional Sequential Kernelized Independence Test): Evaluates global importance (e.g., "How important are stripes for the 'Tiger' class overall?").
- X-SKIT (Explanation Randomization SKIT): Evaluates local importance (e.g., "Why did the model think this specific bird is a Blue Jay?").
Key Insights
- Independence $\neq$ Orthogonality: The authors prove that just because a model's weights are mathematically "orthogonal" to a concept doesn't mean the model is ignoring it. This highlights why simple geometric projections fail and statistical testing is necessary.
- Adaptivity: The betting framework adapts to the hardness of the problem. Highly important concepts are identified almost instantly, while subtle ones require more data.
- Model Agnostic: The framework probes opaque models directly using their predictions, eliminating the need to retrain or rely on linear surrogates.
- Consistency Across Models: Surprisingly, different Vision-Language models (like CLIP vs. BLIP) tend to rank concept importance very similarly, suggesting they learn a shared semantic structure of the world.
Key Results
The framework was evaluated on synthetic datasets, AwA2 (animals), CUB-200-2011 (birds), and Imagenette using various Vision-Language models (CLIP, OpenClip, FLAVA, ALIGN, BLIP).
- Higher Accuracy & Agreement: On the AwA2 dataset, the SKIT method achieved better rank agreement and $f_1$ scores compared to state-of-the-art Post-hoc Concept Bottleneck Models (PCBMs).
- Precise Local Explanations: On the CUB bird dataset, X-SKIT achieved an $f_1$ score of 0.93 in identifying which concepts were actually present and important in specific images.
- Statistical Reliability: Across all experiments, the framework successfully maintained the "Type I error" rate (the probability of a false alarm) at or below the user-defined level (usually 0.05).
Limitations and Future Work
While powerful, the method relies on a "sampler" to estimate the distribution of concepts; biased samplers could compromise the statistical guarantees. Additionally, the framework assumes concepts are already well-represented in the model's feature space—it cannot force a model to use a concept it hasn't learned. Finally, kernel-based tests can be computationally expensive ($O(\tau^2)$ complexity) as sample sizes grow, presenting an opportunity for future work to scale these tests to massive concept banks.
Summary updated