Valid P-Value for Deep Learning-driven Salient Region

Miwa Daiki, Vo Nguyen Le Duy, Ichiro Takeuchi

Introduction

Saliency maps have become a standard tool for interpreting the predictions of deep neural networks (DNNs), highlighting the specific regions of an input that most influenced a model's decision. However, because these maps are generated by the same complex models making the predictions, it is often difficult to determine how reliable the highlighted regions actually are. This paper introduces a rigorous statistical framework to calculate mathematically valid $p$-values for these salient regions, allowing researchers to distinguish between a true discovery and a "cherry-picked" artifact.

The Problem: Selection Bias in Saliency Maps

While saliency maps help peer into the "black box" of deep learning, they suffer from a critical "fragility" problem: selection bias.

If a model is tasked with finding a "salient" region, it will invariably highlight something—even in an image composed entirely of random noise. Standard statistical tests fail in this scenario because they do not account for the fact that the region was specifically chosen by the model based on the data itself. Consequently, "naive" $p$-values calculated without correcting for this bias are almost always near zero. This leads to a dangerously high rate of false positives, where random noise or background artifacts are mistaken for meaningful features.

Methodology: Conditional Selective Inference

To bridge the gap between deep learning and rigorous statistics, the authors employ a framework known as Conditional Selective Inference (SI).

The Core Concept

Instead of asking the standard question, "Is this region significant?", the SI framework asks, "Given that the model selected this specific region as salient, is the signal within it still statistically significant?" By conditioning the probability on the selection event itself, the method effectively corrects for the inherent selection bias.

CNNs as Piecewise Linear Functions

The mathematical breakthrough enabling this approach for DNNs relies on the observation that most modern architectures (utilizing ReLUs, max-pooling, etc.) are fundamentally piecewise linear. The entire network can be conceptualized as a massive collection of linear functions joined at specific boundaries. By treating a Convolutional Neural Network (CNN) as a piecewise linear function, the authors transform a complex "black box" problem into a solvable geometric one.

Auto-Conditioning Framework

To make this practical, the authors developed a Keras-based "Auto-Conditioning" framework. Similar to automatic differentiation, this framework tracks the specific "linear piece" the data occupies as it flows through the network. This tracking identifies the exact range of data values that would result in the same salient region being selected, which is the mathematical key to calculating a valid, truncated $p$-value. Importantly, this modular implementation allows researchers to apply these tests to various CNN architectures without manually deriving the math for every new layer.

Experimental Validation and Results

The researchers evaluated their method using both synthetic datasets (to test theoretical limits) and real-world medical imaging.

Controlling False Positives

In experiments using 1,000 "null" images of pure noise, the proposed SI method successfully controlled the False Positive Rate (FPR) at the target significance level ($\alpha = 0.05$). In stark contrast, the naive approach failed completely, identifying "significant" regions in almost every single noise image. Furthermore, compared to traditional bias-correction methods like the Bonferroni correction, the SI method demonstrated much higher statistical power, meaning it was far better at identifying actual salient features when they truly existed.

Real-World Application: Brain Tumor MRI

The practical value of the method was tested on a dataset of MRI scans containing images both with and without brain tumors.

Comparison of naive and selective p-values on brain MRI scans. The selective p-value correctly identifies the false positive in the healthy brain (top) while confirming the true tumor (bottom).

As demonstrated in the results, the method successfully distinguished between actual tumors and model artifacts. For healthy brains where the model mistakenly highlighted a region, the SI method correctly assigned high $p$-values (non-significant), preventing a false positive diagnosis. Conversely, for actual tumors, the $p$-values remained very small, statistically confirming the discovery.

Limitations and Future Work

The primary limitation of the current method is its reliance on strict alignment. The two-sample test requires a "reference image" (e.g., a scan of a healthy brain) that is perfectly aligned with the query image. If a patient's head is tilted differently between scans, the test may yield inaccurate results.

Future work aims to introduce an automated matching step. By using linear inequalities to find the most appropriate corresponding region in a reference database before running the statistical test, the tool could become significantly more robust for clinical environments where perfectly aligned reference pairs are rare.

Summary updated

Abstract

Various saliency map methods have been proposed to interpret and explain predictions of deep learning models. Saliency maps allow us to interpret which parts of the input signals have a strong influence on the prediction results. However, since a saliency map is obtained by complex computations in deep learning models, it is often difficult to know how reliable the saliency map itself is. In this study, we propose a method to quantify the reliability of a saliency region in the form of p-values. Our idea is to consider a saliency map as a selected hypothesis by the trained deep learning model and employ the selective inference framework. The proposed method provably provides a valid p-value for the detected salient region, i.e., we can provably control the false positive rate of the detected salient region. We demonstrate the validity of the proposed method through numerical examples in synthetic and real datasets. Furthermore, we develop a Keras-based framework for conducting the proposed selective inference for a wide class of CNNs without additional implementation cost.
BibTeX
@inproceedings{daiki2023valid,
  author = {Miwa Daiki and Vo Nguyen Le Duy and Ichiro Takeuchi},
  title = {{Valid P-Value for Deep Learning-driven Salient Region}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=qihMOPw4Sf_},
}
Figure