Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis

Benjamin Lambert, Florence Forbes, Senan Doyle, Harmonie Dehaene, Michel DojatArtif. Intell. Medicine 2024 · DOI

TL;DR

This paper provides a comprehensive review of uncertainty quantification (UQ) methods for deep learning in medical image analysis, addressing the critical need for trustworthy AI in clinical settings. It categorizes existing UQ techniques, discusses their applications, and highlights the challenges and future directions for making AI models more reliable and interpretable for clinicians.

What Problem Does This Paper Address?

Deep learning (DL) models have shown immense promise in medical image analysis, but their widespread adoption by clinicians is hindered by a lack of trust. This distrust stems from the "black-box" nature of these models, their tendency to be overconfident in their predictions (even when incorrect), and their susceptibility to adversarial attacks. Clinicians need to understand not just what a model predicts, but also how confident it is in that prediction. This is crucial for identifying potentially erroneous diagnoses, flagging unusual cases, and ensuring patient safety. Uncertainty quantification (UQ) aims to provide these confidence estimates, making DL models more interpretable and trustworthy for clinical use.

Methods

The paper reviews a wide array of UQ methods applied to medical image analysis, categorizing them into several frameworks:

  • Softmax Probabilities: The most intuitive approach, using the output probabilities of a DL model as an indicator of uncertainty. However, modern DL models are often poorly calibrated, making these raw probabilities unreliable.
  • Conformal Prediction (CP): A statistical framework that provides provable guarantees on prediction coverage, generating predictive sets or intervals that are guaranteed to contain the true label with a user-defined confidence.
  • Bayesian Neural Networks (BNNs): These models treat network weights as probability distributions, allowing for the modeling of epistemic uncertainty (uncertainty due to lack of knowledge).
  • Monte Carlo Dropout (MC Dropout): A computationally efficient approximation of Bayesian inference, where dropout is kept active during inference to generate multiple predictions and estimate uncertainty.
  • Ensemble Methods: Combining predictions from multiple independently trained models to improve robustness and estimate uncertainty.
  • Learned Uncertainty Frameworks: Methods that directly learn to predict uncertainty alongside the main task, often by modeling data noise (aleatoric uncertainty).
  • Test-Time Augmentation (TTA): Generating multiple augmented versions of an input image at inference time and analyzing the variability in predictions to estimate uncertainty.
  • Generative Models: Using models like Probabilistic U-Nets to generate multiple plausible segmentations and assess uncertainty based on their diversity.
  • Feature-Based Methods: Detecting out-of-distribution (OOD) samples by analyzing intermediate feature representations, as OOD samples are expected to yield higher epistemic uncertainty.
  • Evidential Deep Learning (EDL): A framework based on Dempster-Shafer Theory, which assigns belief masses to classes and an overall uncertainty mass, allowing for the modeling of both aleatoric and epistemic uncertainty.

The paper also discusses the crucial aspect of structural uncertainty, which aims to scale voxel-level uncertainty estimates to lesion-level or case-level scores, making them more clinically relevant.

Datasets

The paper does not introduce new datasets. Instead, it reviews existing literature that utilizes various medical imaging datasets for tasks like classification and segmentation. These datasets would typically include modalities such as MRI, CT, Ultrasound, and histopathological images. The specific datasets used in the reviewed papers are detailed in the appendices.

Key Insights and Ideas

  • UQ is Essential for Clinical Trust: The paper strongly emphasizes that for DL models to be accepted in clinical practice, they must provide not just accurate predictions but also reliable estimates of their confidence. This is the core driver for UQ research in this domain.
  • MC Dropout and Ensembles Dominate: MC Dropout and Ensemble methods are the most frequently implemented UQ techniques in medical imaging literature, likely due to their relative ease of implementation and effectiveness in capturing both aleatoric and epistemic uncertainties.
  • Bridging Voxel-Level to Clinical Relevance: A significant challenge is translating low-level (voxel-wise) uncertainty estimates into clinically meaningful scores at the lesion or patient (case) level. Structural uncertainty frameworks are key to this.
  • OOD Detection is Crucial but Under-Researched: Detecting when a model encounters data significantly different from its training distribution (Out-of-Distribution detection) is vital for safety in real-world clinical scenarios, yet it is an area with relatively few evaluation protocols.
  • Calibration is Key for Softmax-Based UQ: While simple, Softmax probabilities are only useful if the model is well-calibrated. The paper highlights the importance of calibration metrics for assessing the reliability of these basic UQ approaches.
  • Bayesian Methods are Under-Utilized: Despite their strong theoretical foundation for UQ, pure Bayesian methods are less commonly implemented due to their complexity and computational cost, with approximations like MC Dropout being preferred.

Key Results

The paper is a review, so it doesn't present new experimental results. Instead, it synthesizes findings from 218 reviewed papers, identifying trends in UQ method implementation and evaluation.

  • Prevalence of UQ Methods: The review found that 67.83% of implemented UQ methods rely on sampling protocols (MC Dropout, Deep Ensembles, BNNs, TTA, Generative Models), which can increase computational burden. Single-step methods like Softmax, Evidential Deep Learning, and Feature-based methods are highlighted as promising for time-critical applications.
  • Uncertainty Types: The majority of implemented UQ methods (61.95%) estimate both aleatoric and epistemic uncertainties, followed by aleatoric-only (32.74%) and epistemic-only (4.13%).
  • Evaluation Protocols: The most common evaluation protocols are Error Detection & Referral (27.89%), Qualitative Assessment (24.30%), and Calibration (17.53%). OOD detection is less frequently evaluated (7.97%).

Limitations and Future Work

  • Computational Cost of Sampling Methods: The reliance on sampling-based UQ methods can significantly increase computational demands, potentially hindering their adoption in real-time clinical pipelines, especially for large 3D medical images.
  • Robustness to Domain Shift: The performance of UQ estimates can degrade significantly in the presence of domain shifts (e.g., different scanners, patient populations). This robustness is crucial for clinical deployment but is not always adequately evaluated.
  • Limited OOD Detection Evaluation: Despite its importance, OOD detection is not as widely evaluated as other UQ aspects, partly due to the difficulty of obtaining representative OOD datasets.
  • Link between Explainability and Uncertainty: The paper suggests that future work should explore the connection between explainability (understanding why a model made a prediction) and uncertainty quantification. This could lead to more interpretable and trustworthy AI systems by helping users understand the sources of uncertainty.
  • Development of Single-Step UQ Methods: Further research into efficient, single-step UQ methods is needed to address the computational limitations of sampling-based approaches.

Summary updated

BibTeX
@article{lambert2024trustworthy,
  author = {Benjamin Lambert and Florence Forbes and Senan Doyle and Harmonie Dehaene and Michel Dojat},
  title = {{Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis}},
  journal = {Artif. Intell. Medicine},
  year = {2024},
  doi = {10.1016/j.artmed.2024.102830},
}
Figure