Abstract
Although black-box models can accurately predict outcomes such as weather patterns, they often lack transparency, making it challenging to extract meaningful insights. Model explanations attempt to identify the essential features of a model, but these explanations can be inconsistent: two near-optimal models may admit vastly different explanations. In this paper, we propose a solution to this problem by constructing uncertainty sets for explanations of the optimal model(s) in both frequentist and Bayesian settings. Our uncertainty sets are guaranteed to include the explanation of the optimal model with high probability, even though this model is unknown. We demonstrate the effectiveness of our approach in both synthetic and real-world experiments, illustrating how our uncertainty sets can be used to calibrate trust in model explanations.
Introduction: The Reliability Gap in XAI
In the world of machine learning, Explainable AI (XAI) tools like SHAP or LIME are typically used for two distinct goals: model auditing (checking if our model is broken) and scientific inquiry (trying to learn something new about the world). However, a significant hurdle remains: model multiplicity. Modern models are often "unstable," meaning many different models can achieve nearly identical accuracy on the same dataset while providing wildly different explanations.
If a researcher explains only a single "best-fitting" model, they risk interpreting a fluke of that specific training run rather than a fundamental truth about the data. This paper addresses the lack of "uncertainty awareness" in XAI, moving beyond explaining a single trained model to explaining the range of models that could have been the right one.
The "Naive" Trap and the Need for Rigor
A common approach to estimate uncertainty is to use "naive" methods like bootstrapping or subsampling a single model. However, as shown in the researchers' experiments, these methods consistently underestimate uncertainty. While they may produce tight intervals, those intervals frequently fail to contain the "true" explanation of the underlying data-generating process.
To address this, the authors propose three rigorous methods to ensure the "optimal" explanation is captured with high probability (e.g., 95%):
- Frequentist (UCEI): This method uses learning theory to identify a set of all models that perform "well enough" on the training data and calculates the most extreme possible explanations within that set.
- Bayesian (BEI): This method uses samples from a model's posterior distribution to build credible intervals for the explanation.
- Conformal (CEI): Inspired by conformal prediction, this method uses a prior distribution to simulate "ground truth" explanations and calibrates a "nonconformity score" to ensure coverage.
Data Volume and Trust Calibration
The size of these uncertainty sets serves as a critical "trust signal." When uncertainty is high, the set for a feature's importance might span from negative to positive, indicating that the data is currently insufficient to determine if that feature is actually helpful or harmful, even if a single "best" model says it is important.
This relationship is clearly visible in experiments with the MNIST handwritten digit dataset. With only 10 training examples, the range of plausible explanations for a digit's classification is massive. As the training set grows to 1,000 examples, the uncertainty intervals shrink significantly, showing how more data leads to more reliable scientific insights.
Key Results and Scientific Caution
In synthetic tests, the proposed Frequentist, Bayesian, and Conformal methods all achieved the target 95% coverage rate. In contrast, the naive method captured the true explanation only about 50% of the time.
On real-world tabular datasets, the framework provides actionable insights:
- MPG Dataset: Features like
displacement,horsepower, andweightshowed high importance with very tight uncertainty intervals, suggesting high confidence in these drivers of fuel efficiency. - PROTEIN Dataset: The resulting intervals were wide and overlapping. Rather than a failure, this is a success of the framework: it correctly signals that the data is too noisy or the models too diverse to make a definitive claim about which protein features are most important.
Conclusion
By shifting the focus from explaining a single model to explaining the set of all plausible models, this framework provides a statistically grounded way to calibrate trust in XAI. While these methods introduce some computational overhead, they represent a vital step toward making machine learning a more reliable tool for scientific discovery and robust model auditing.
Summary updated