Introduction
Saliency maps have become a standard tool for interpreting the predictions of deep neural networks (DNNs), highlighting the specific regions of an input that most influenced a model's decision. However, because these maps are generated by the same complex models making the predictions, it is often difficult to determine how reliable the highlighted regions actually are. This paper introduces a rigorous statistical framework to calculate mathematically valid $p$-values for these salient regions, allowing researchers to distinguish between a true discovery and a "cherry-picked" artifact.
The Problem: Selection Bias in Saliency Maps
While saliency maps help peer into the "black box" of deep learning, they suffer from a critical "fragility" problem: selection bias.
If a model is tasked with finding a "salient" region, it will invariably highlight something—even in an image composed entirely of random noise. Standard statistical tests fail in this scenario because they do not account for the fact that the region was specifically chosen by the model based on the data itself. Consequently, "naive" $p$-values calculated without correcting for this bias are almost always near zero. This leads to a dangerously high rate of false positives, where random noise or background artifacts are mistaken for meaningful features.
Methodology: Conditional Selective Inference
To bridge the gap between deep learning and rigorous statistics, the authors employ a framework known as Conditional Selective Inference (SI).
The Core Concept
Instead of asking the standard question, "Is this region significant?", the SI framework asks, "Given that the model selected this specific region as salient, is the signal within it still statistically significant?" By conditioning the probability on the selection event itself, the method effectively corrects for the inherent selection bias.
CNNs as Piecewise Linear Functions
The mathematical breakthrough enabling this approach for DNNs relies on the observation that most modern architectures (utilizing ReLUs, max-pooling, etc.) are fundamentally piecewise linear. The entire network can be conceptualized as a massive collection of linear functions joined at specific boundaries. By treating a Convolutional Neural Network (CNN) as a piecewise linear function, the authors transform a complex "black box" problem into a solvable geometric one.
Auto-Conditioning Framework
To make this practical, the authors developed a Keras-based "Auto-Conditioning" framework. Similar to automatic differentiation, this framework tracks the specific "linear piece" the data occupies as it flows through the network. This tracking identifies the exact range of data values that would result in the same salient region being selected, which is the mathematical key to calculating a valid, truncated $p$-value. Importantly, this modular implementation allows researchers to apply these tests to various CNN architectures without manually deriving the math for every new layer.
Experimental Validation and Results
The researchers evaluated their method using both synthetic datasets (to test theoretical limits) and real-world medical imaging.
Controlling False Positives
In experiments using 1,000 "null" images of pure noise, the proposed SI method successfully controlled the False Positive Rate (FPR) at the target significance level ($\alpha = 0.05$). In stark contrast, the naive approach failed completely, identifying "significant" regions in almost every single noise image. Furthermore, compared to traditional bias-correction methods like the Bonferroni correction, the SI method demonstrated much higher statistical power, meaning it was far better at identifying actual salient features when they truly existed.
Real-World Application: Brain Tumor MRI
The practical value of the method was tested on a dataset of MRI scans containing images both with and without brain tumors.
As demonstrated in the results, the method successfully distinguished between actual tumors and model artifacts. For healthy brains where the model mistakenly highlighted a region, the SI method correctly assigned high $p$-values (non-significant), preventing a false positive diagnosis. Conversely, for actual tumors, the $p$-values remained very small, statistically confirming the discovery.
Limitations and Future Work
The primary limitation of the current method is its reliance on strict alignment. The two-sample test requires a "reference image" (e.g., a scan of a healthy brain) that is perfectly aligned with the query image. If a patient's head is tilted differently between scans, the test may yield inaccurate results.
Future work aims to introduce an automated matching step. By using linear inequalities to find the most appropriate corresponding region in a reference database before running the statistical test, the tool could become significantly more robust for clinical environments where perfectly aligned reference pairs are rare.
Summary updated