Bayesian uncertainty estimation for detection of long-tailed and unseen conditions in medical images

Mina Rezaei, Janne J. Näppi, Bernd Bischl, Hiroyuki YoshidaJournal of Medical Imaging 2023 · DOI

TL;DR

This paper introduces a novel Bayesian ensemble learning framework that leverages uncertainty estimation to tackle the challenge of imbalanced datasets in medical imaging. By effectively identifying and adapting to rare "tail" classes and out-of-distribution samples, their method significantly improves performance in tasks like semantic segmentation and disease detection, paving the way for more robust AI in healthcare.

What Problem Does This Paper Address?

Deep learning models have shown great promise in medical imaging, but they often struggle with real-world data. A major hurdle is the "long-tailed" distribution of medical datasets, where common conditions (head classes) are abundant, but rare diseases or anomalies (tail classes) are severely underrepresented. This imbalance makes it difficult for models to learn accurate decision boundaries for these infrequent cases. Furthermore, models trained on one distribution can perform poorly when encountering data from a different, unseen distribution (out-of-distribution or OOD detection). Existing methods for handling imbalanced data often lead to overfitting or require careful manual tuning. This paper addresses the critical need for deep learning systems that can automatically identify and adapt to these real-world imbalanced and OOD scenarios in medical imaging.

Methods

The core of the proposed solution is a deep Bayesian ensemble learning framework. This framework combines several key ideas:

  • Bayesian Uncertainty Estimation: Instead of just predicting a class label, the model estimates the uncertainty associated with its predictions. The intuition is that rare classes or OOD samples will naturally exhibit higher uncertainty.
  • Dynamic Monte-Carlo (MC) Dropout: This technique is used to approximate Bayesian inference. During training, random subsets of neurons are "dropped out" with a probability that can dynamically change. This allows the model to generate multiple different predictions for the same input, effectively creating an ensemble of models. The variance in these predictions serves as the uncertainty estimate.
  • Ensemble of Networks: The framework utilizes a generator and multiple discriminator networks, forming a type of Generative Adversarial Network (GAN). This ensemble approach, combined with dynamic MC dropout, helps to boost predictive performance and improve robustness.
  • Uncertainty-Aware Adaptation: The model leverages the estimated uncertainties to adapt its decision boundaries. By recognizing higher uncertainty for tail classes, it can expand the boundaries to better capture these infrequent cases, leading to improved generalization.
  • Combination of Losses: The framework employs a combination of losses, including adversarial losses, to train the generator and discriminators effectively.

Datasets

The authors evaluated their framework on four publicly available medical imaging datasets:

  • LiTS (Liver Tumor Segmentation): Used for semantic segmentation of liver tumors, featuring a significant class imbalance between normal tissue and small lesions.
  • CHAOS (Combined Healthy Abdominal Organ Segmentation): Used for multi-organ semantic segmentation on both CT and MR images, also with imbalanced class distributions.
  • Glaucoma Detection Dataset: A real-world dataset for detecting glaucoma from retinal images, with an imbalance ratio of 1:30.
  • REFUGE 2020: Used for Out-of-Distribution (OOD) detection, specifically to test the model's ability to identify samples from a different distribution than the training data.

Key Insights and Ideas

  • Uncertainty as a Signal for Rarity: The fundamental insight is that rare classes in imbalanced datasets inherently exhibit higher predictive uncertainty. This is a powerful signal that can be directly exploited by the model.
  • Bridging the Gap to OOD: By framing the detection of infrequent samples as an OOD problem, the uncertainty estimation approach can naturally extend to identifying entirely novel, unseen data distributions.
  • Ensemble Power for Robustness: The combination of Bayesian approximation (via MC dropout) and a GAN-based ensemble structure creates a highly robust system that benefits from diverse perspectives.
  • Dynamic Adaptation is Key: The "dynamic" aspect of the MC dropout and the adaptive nature of the uncertainty-aware decision boundaries are crucial for handling the complexities of long-tailed and OOD data.

Key Results

The proposed Bayesian ensemble GAN demonstrated significant improvements across all tested tasks:

  • Semantic Segmentation (LiTS & CHAOS): The framework achieved significantly higher performance (p < 0.01 and p < 0.001 respectively) compared to comparator models like Conditional GAN, U-Net, and standard Ensemble GANs, across metrics such as ASSD, F1 score, precision, and recall. For instance, on the LiTS dataset, it achieved an F1 score of 96.2 ± 0.2.
  • In-Domain Generalization (Glaucoma Detection): The model achieved the best classification performance in terms of F1 score (87.8 ± 0.1 with 10 members) and the second-best in expected calibration error (ECE). This indicates strong generalization within the same data distribution.
  • OOD Detection (Glaucoma vs. REFUGE): The Bayesian ensemble GAN significantly outperformed comparator models in AUROC scores for OOD detection (p < 0.01), achieving a score of 79.43 ± 0.1 with 10 members.

Limitations and Future Work

The authors acknowledge the following limitations:

  • Computational Expense: Training GANs, especially large ensembles, is computationally expensive and time-consuming, particularly for large datasets.
  • Memory Intensive: Storing multiple models for the ensemble can be memory-intensive, especially if the individual models are large.

These limitations suggest avenues for future work, such as exploring more efficient training strategies for GANs and developing methods for model compression or more memory-efficient ensemble representations.

Summary updated

Abstract

Training of the proposed deep Bayesian ensemble learning framework with dynamic Monte-Carlo dropout and a combination of losses yielded the best generalization to unseen samples from imbalanced medical imaging datasets across different learning tasks.
BibTeX
@article{rezaei2023bayesian,
  author = {Mina Rezaei and Janne J. Näppi and Bernd Bischl and Hiroyuki Yoshida},
  title = {{Bayesian uncertainty estimation for detection of long-tailed and unseen conditions in medical images}},
  journal = {Journal of Medical Imaging},
  year = {2023},
  doi = {10.1117/1.jmi.10.5.054501},
}
Figure