Testing Semantic Importance via Betting

Jacopo Teneggi, Jeremias SulamNeurIPS 2024 · OpenReview

Explainable AI (XAI) often focuses on highlighting important pixels, but humans reason in concepts. A doctor diagnosing an X-ray doesn't look for "Pixel (242, 11)"; they look for a "nodule" or "calcification." While recent methods like Concept Bottleneck Models and Concept Activation Vectors (CAVs) attempt to bridge this gap, they frequently lack the statistical rigor needed for high-stakes domains.

This paper introduces a rigorous statistical framework for understanding why "black-box" machine learning models make specific decisions. By focusing on semantic concepts—human-understandable ideas like "furry," "stripes," or "wings"—the authors provide a way to rank concept importance with formal statistical guarantees.

TL;DR

The authors propose measuring the importance of semantic concepts in predictive models using a "Testing by Betting" approach. By framing importance as a conditional independence test and using sequential e-values, they can rank how much a concept (e.g., "whiskers") influences a model's prediction (e.g., "cat"), providing rigorous control over false positives.

Overview of the Testing by Betting framework for ranking semantic concepts.

The Problem with Current Explanations

Existing concept-based methods usually cannot quantify the probability that a concept is wrongly identified as important (the False Positive Rate). Furthermore, many approaches require training a simpler "surrogate" model to explain the complex one, which might not accurately reflect how the original black box operates. This paper aims to provide statistical guarantees—like Type I error and False Discovery Rate control—directly on the original, non-linear model without retraining.

Methods: Testing by Betting

The core innovation treats semantic importance as a Conditional Independence problem. If a model’s prediction ($Y$) is independent of a concept ($Z_j$) given all other concepts ($Z_{-j}$), then that concept isn't contributing unique information to the decision.

To test this, the authors utilize Testing by Betting:

  1. The Game: Imagine a "bettor" starting with $1. At each step, they observe a new data point and place a wager against the null hypothesis (which states the concept is unimportant).
  2. Wealth Process: If the concept is truly important, the bettor’s wealth grows. If it reaches a threshold (e.g., $1/\alpha$, where $\alpha$ is the significance level), the null hypothesis is rejected, and the concept is deemed "important."
  3. Sequential Testing: Unlike traditional p-values that require a fixed dataset size, this betting approach is sequential and stops as soon as it gathers enough evidence. This naturally creates a rank of importance: concepts that make the bettor "rich" the fastest are the most critical.

The authors introduce two specific algorithms:

  • C-SKIT (Conditional Sequential Kernelized Independence Test): Evaluates global importance (e.g., "How important are stripes for the 'Tiger' class overall?").
  • X-SKIT (Explanation Randomization SKIT): Evaluates local importance (e.g., "Why did the model think this specific bird is a Blue Jay?").

Key Insights

  • Independence $\neq$ Orthogonality: The authors prove that just because a model's weights are mathematically "orthogonal" to a concept doesn't mean the model is ignoring it. This highlights why simple geometric projections fail and statistical testing is necessary.
  • Adaptivity: The betting framework adapts to the hardness of the problem. Highly important concepts are identified almost instantly, while subtle ones require more data.
  • Model Agnostic: The framework probes opaque models directly using their predictions, eliminating the need to retrain or rely on linear surrogates.
  • Consistency Across Models: Surprisingly, different Vision-Language models (like CLIP vs. BLIP) tend to rank concept importance very similarly, suggesting they learn a shared semantic structure of the world.

Key Results

The framework was evaluated on synthetic datasets, AwA2 (animals), CUB-200-2011 (birds), and Imagenette using various Vision-Language models (CLIP, OpenClip, FLAVA, ALIGN, BLIP).

Quantitative performance and global concept importance rankings.
  • Higher Accuracy & Agreement: On the AwA2 dataset, the SKIT method achieved better rank agreement and $f_1$ scores compared to state-of-the-art Post-hoc Concept Bottleneck Models (PCBMs).
  • Precise Local Explanations: On the CUB bird dataset, X-SKIT achieved an $f_1$ score of 0.93 in identifying which concepts were actually present and important in specific images.
  • Statistical Reliability: Across all experiments, the framework successfully maintained the "Type I error" rate (the probability of a false alarm) at or below the user-defined level (usually 0.05).

Limitations and Future Work

While powerful, the method relies on a "sampler" to estimate the distribution of concepts; biased samplers could compromise the statistical guarantees. Additionally, the framework assumes concepts are already well-represented in the model's feature space—it cannot force a model to use a concept it hasn't learned. Finally, kernel-based tests can be computationally expensive ($O(\tau^2)$ complexity) as sample sizes grow, presenting an opportunity for future work to scale these tests to massive concept banks.

Summary updated

Abstract

Recent works have extended notions of feature importance to semantic concepts that are inherently interpretable to the users interacting with a black-box predictive model. Yet, precise statistical guarantees such as false positive rate and false discovery rate control are needed to communicate findings transparently, and to avoid unintended consequences in real-world scenarios. In this paper, we formalize the global (i.e., over a population) and local (i.e., for a sample) statistical importance of semantic concepts for the predictions of opaque models by means of conditional independence, which allows for rigorous testing. We use recent ideas of sequential kernelized independence testing to induce a rank of importance across concepts, and we showcase the effectiveness and flexibility of our framework on synthetic datasets as well as on image classification using several vision-language models.
BibTeX
@inproceedings{teneggi2024testing,
  author = {Jacopo Teneggi and Jeremias Sulam},
  title = {{Testing Semantic Importance via Betting}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=A0HSmrwtLH},
}
Figure