I Bet You Did Not Mean That: Testing Semantic Importance via Betting

Jacopo Teneggi, Jeremias SulamarXiv 2024 · DOI · arXiv

Abstract

Explainable AI (XAI) often struggles to bridge the gap between low-level input features and high-level human concepts. This paper introduces a rigorous statistical framework to determine which semantic concepts (e.g., "stripes," "wings") are truly influential in a black-box model's predictions. By framing importance as a sequential "game of betting," the authors provide a method to rank concepts with formal guarantees on false positives, bypassing the need for simplified surrogate models.

Introduction

Most XAI tools focus on the input space, highlighting pixels or words. However, humans reason using concepts—a radiologist looks for "spiculation," not just pixel clusters. While Concept Bottleneck Models (CBMs) address this, they often rely on linear surrogates that may not faithfully represent the original model. This work formalizes semantic importance through the lens of conditional independence testing, providing precise statistical guarantees.

Methodology

The authors propose testing importance at three granularities:

  1. Global Importance (SKIT): Population-wide dependence.
  2. Global Conditional Importance (c-SKIT): Unique information not captured by other concepts.
  3. Local Conditional Importance (x-SKIT): Importance for a specific individual prediction.

Testing by Betting

Instead of traditional p-values, the framework uses Testing by Betting. A "bettor" wagers against a null hypothesis (e.g., "this concept is not important"). If the concept is important, the bettor accumulates wealth (an e-value). Rejection occurs when wealth crosses a threshold ($1/\alpha$). This sequential approach is data-efficient and allows for ranking concepts based on rejection time.

Technical Implementation

The framework utilizes Maximum Mean Discrepancy (MMD) to measure distances between probability distributions. It includes a greedy post-processor to control the False Discovery Rate (FDR) when testing multiple concepts simultaneously.

Experimental Evaluation

The framework was tested on synthetic data, Counting MNIST, and real-world datasets like Animals with Attributes 2 (AwA2) and CUB-200-2011.

Concept importance results for various bird species using CLIP:ViT-L/14. Green checkmarks indicate concepts identified as important by the framework, while red crosses indicate unimportant ones.

Key Results

  • High Fidelity: On AwA2, the framework achieved 99.5% accuracy in identifying important concepts, significantly outperforming standard Concept Bottleneck Models (~95%).
  • Model Agnosticism: Importance ranks were stable across different vision-language models (CLIP, OpenClip, BLIP), suggesting shared semantic strategies.
  • Local Insights: The x-SKIT test successfully identified concepts relevant to specific images (e.g., "fishing" for a dog in water) that weren't globally relevant.

Discussion and Limitations

The framework's power depends on the quality of conditional samplers and kernel selection. While non-parametric KDE-based samplers were used, high-dimensional spaces remain a challenge. Additionally, the method tests the provided concepts; it does not discover new ones.

Conclusion

This betting-based approach offers a robust, statistically grounded way to interpret black-box models through human-understandable concepts. By avoiding surrogates and providing formal guarantees, it sets a new standard for reliable semantic explanations in AI.

Summary updated

Abstract

Recent works have extended notions of feature importance to semantic concepts that are inherently interpretable to the users interacting with a black-box predictive model. Yet, precise statistical guarantees, such as false positive rate and false discovery rate control, are needed to communicate findings transparently and to avoid unintended consequences in real-world scenarios. In this paper, we formalize the global (i.e., over a population) and local (i.e., for a sample) statistical importance of semantic concepts for the predictions of opaque models by means of conditional independence, which allows for rigorous testing. We use recent ideas of sequential kernelized independence testing (SKIT) to induce a rank of importance across concepts, and showcase the effectiveness and flexibility of our framework on synthetic datasets as well as on image classification tasks using several and diverse vision-language models.
BibTeX
@misc{teneggi2024i,
  author = {Jacopo Teneggi and Jeremias Sulam},
  title = {{I Bet You Did Not Mean That: Testing Semantic Importance via Betting}},
  year = {2024},
  eprint = {2405.19146},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2405.19146},
}
Figure