Quantifying Uncertainty in Machine Learning Explanations

267 concepts175 papers4 proposals

Imagine a medical AI diagnostic tool that flags a patient's X-ray as malignant. To justify its decision, it generates a saliency map highlighting a specific region of the lung. The explanation looks authoritative and definitive.

But what if the explanation algorithm itself is highly unstable, and running it a second time would highlight an entirely different, healthy part of the lung? Or worse, what if the underlying model is completely unsure about its prediction, yet the explanation interface generates a highly persuasive justification anyway?

Traditional Explainable AI (XAI) methods—like LIME, SHAP, or counterfactuals—typically present feature attributions as absolute truths. This creates a dangerous illusion of certainty. To bridge this gap, a growing body of research focuses on uncertainty-aware interpretability. This paradigm operates on two fronts: quantifying the uncertainty of the explanation itself (giving error bars to our explanations) and explaining the root causes of a model's predictive uncertainty (explaining why a model is unsure).


Putting Error Bars on Explanations

Why are explanations uncertain? Popular local surrogate methods approximate a complex model's behavior by sampling points around an input. This sampling introduces stochastic noise. As Hyunseung Hwang et al. demonstrate in Explanation Multiplicity in SHAP: Characterization and Assessment, this randomness can lead to significant "rank-order churn," where feature importance rankings shift across repeated runs. Furthermore, Jonas Schulz et al. note in Uncertainty Quantification of Surrogate Explanations: an Ordinal Consensus Approach that standard surrogates often assume the base model is confident, ignoring alternative predictive paths that the model might be considering.

To capture this volatility, researchers rely on two primary mathematical frameworks: Conformal Prediction and Bayesian Inference.

Conformalized Explanations

Conformal prediction provides a way to bound explanation errors with rigorous statistical guarantees. Instead of returning a single, potentially misleading importance score, a conformalized explainer outputs a prediction interval.

For instance, Amr Alkhatib et al. in Approximating Score-based Explanation Techniques Using Conformal Regression use inductive conformal regression to place mathematically guaranteed error bars around approximated SHAP scores. The intuition is elegant: the width of the interval serves as an inherent reliability metric. If the interval is excessively wide, it signals that the surrogate model lacks sufficient information to provide a trustworthy explanation, warning the user to treat the attribution with caution.

Bayesian Surrogates

Instead of calculating a single point estimate for feature importance, Bayesian approaches treat explanation parameters as probability distributions. In BayLIME: Bayesian local interpretable model-agnostic explanations, Xingyu Zhao et al. introduce a Bayesian framework to LIME. By calculating the posterior precision of feature weights, the system explicitly communicates its confidence level to the user. If the posterior variance is high, the explanation is flagged as unreliable, preventing users from over-relying on noisy local approximations.


Explaining Why a Model is Unsure

The second facet of this research direction flips the question: when a model outputs a high-uncertainty prediction, why is it confused? Is it because the input is out-of-distribution (OOD), or because the features are inherently ambiguous?

A powerful way to build intuition here is through Counterfactual Latent Uncertainty Explanations (CLUE). Introduced in Getting a CLUE: A Method for Explaining Uncertainty Estimates, the authors seek the smallest, most realistic change required to make a model confident.

[Uncertain Input] ───(Latent Space Optimization)───> [Confident Counterfactual]
       │                                                      │
       └───────────────────[ Difference = CLUE ]──────────────┘

Imagine a model that is uncertain about a patient's diagnosis. CLUE optimizes the input features within a generative model's latent space to find a nearby, realistic synthetic patient for whom the model is certain. The delta between the real patient and the certain counterfactual highlights exactly which features triggered the model's doubt.

Building on this concept, Dan Ley et al. in Diverse, Global and Amortised Counterfactual Explanations for Uncertainty Estimates expand this framework to generate diverse sets of counterfactual recourses. This allows users to see multiple distinct paths to certainty, which is crucial when navigating regions with sparse training data or outliers.


Calibrating Human Trust

Why does quantifying explanation uncertainty matter so deeply? The ultimate goal is trust calibration.

When an AI provides a highly convincing natural language explanation for an incorrect prediction, humans suffer from automation bias. In a clinical study, Maxime Kayser et al. in Fool Me Once? Contrasting Textual and Visual Explanations in a Clinical Decision-Support Setting found that persuasive text-based justifications frequently led clinicians to blindly trust erroneous AI advice.

By integrating uncertainty quantification directly into the explanation interface—whether through visual confidence masks, interval bounds, or explicit verbalized uncertainty markers—systems can break this illusion. When an explanation explicitly states its own limitations, human experts are prompted to step in, override the system, and verify the results manually.

History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_explanation_uncertainty_quantification_3e3a1e87,
  title = {explanation uncertainty quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-uncertainty-quantification/rdr_3e3a1e8700e44f30ab038919c68be9cd},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_explanation_uncertainty_quantification_3e3a1e87_v1,
  title = {explanation uncertainty quantification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/explanation-uncertainty-quantification/rdr_3e3a1e8700e44f30ab038919c68be9cd/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake