Navigating Uncertainty in Medical Image Segmentation

Kilian Zepf, Jes Frellsen, Aasa Feragen

Abstract We address the selection and evaluation of uncertain segmentation methods in medical imaging and present two case studies: prostate segmentation, illustrating that for minimal annotator variation simple deterministic models can suffice, and lung lesion segmentation, highlighting the limitations of the Generalized Energy Distance (GED) in model selection. Our findings lead to guidelines for accurately choosing and developing uncertain segmentation models, that integrate aleatoric and epistemic components. These guidelines are designed to aid researchers and practitioners in better developing, selecting, and evaluating uncertain segmentation methods, thereby facilitating enhanced adoption and effective application of segmentation uncertainty in practice.

Introduction

In medical artificial intelligence, "uncertainty" is frequently employed as a catch-all term, yet not all uncertainty is created equal. To build reliable clinical tools, researchers must distinguish between aleatoric uncertainty—the inherent randomness in data, such as differing expert opinions on a tumor’s edge—and epistemic uncertainty, which stems from the model’s lack of knowledge regarding unfamiliar data.

This research provides a critical reality check for uncertainty quantification (UQ) in medical image segmentation. The authors identify two primary pitfalls in current practices: the over-engineering of models for simple noise and the reliance on flawed evaluation metrics that conflate different types of error. By analyzing these issues, the study offers a roadmap for more effective model selection and evaluation.

The Problem: Over-Engineering and Flawed Evaluation

Current research often falls into two traps. First, there is a tendency toward over-engineering, where heavy probabilistic machinery is used to model simple "drawing noise" (unsystematic variation like shaky hands) that a standard deterministic model could handle. Second, the field relies heavily on the Generalized Energy Distance (GED) to rank models. GED is problematic because it conflates the difficulty of detecting a lesion with the difficulty of delineating its boundaries. This lack of granularity can lead to misleading conclusions about which model architecture is truly superior for a specific clinical task.

Methodology

The authors compared several popular uncertainty-aware architectures against a standard deterministic U-Net:

  • Ensembles: Training multiple models to measure disagreement.
  • Probabilistic U-Net: Utilizing latent variables to capture multiple possible segmentation outputs.
  • Stochastic Segmentation Networks (SSN): Modeling logit distributions to capture spatial correlations.
  • MC-Dropout: A common technique for estimating epistemic uncertainty through stochastic passes during inference.

To address the limitations of GED, the authors proposed decomposing it into two task-specific metrics:

  1. $D^2_{Det}$ (Detection GED): Measures how well the model captures the ambiguity of whether an object exists at all.
  2. $D^2_{IoU}$ (Segmentation GED): Measures variation in boundary shapes, specifically excluding cases where the object was missed entirely.

Experimental Design

The study utilized two distinct benchmarks to illustrate different types of uncertainty:

  • QUBIQ 2021 (Prostate): A dataset featuring six annotators. Variation here is minimal, consisting mostly of unsystematic drawing errors.
  • LIDC-IDRI (Lung Lesions): A highly ambiguous dataset where four radiologists often disagree on whether a lesion exists, as well as its exact shape.

Results and Discussion

The "Softmax is Enough" Revelation

In the prostate segmentation task, where expert agreement is high, the entropy from a simple U-Net’s softmax output provided uncertainty estimates that were as effective as complex aleatoric models. In these instances, adding complexity increases computational overhead without providing additional clinical value. The deterministic U-Net and SSN performed on par, while Ensembles and Probabilistic U-Nets tended to overestimate the variation in the data.

Disentangling Detection and Segmentation

In the lung lesion task, no single model emerged as a universal winner. Ensembles proved best at capturing detection ambiguity ($D^2_{Det}$), but other models were superior at boundary delineation ($D^2_{IoU}$). This confirms that aggregate scores like the original GED can mask a model's true strengths and weaknesses. If a model is excellent at finding lesions but poor at defining their edges, a single GED score might rank it similarly to a model with the opposite profile.

The Epistemic Gatekeeper

The authors propose a "two-step" framework for UQ. Epistemic uncertainty should serve as a gatekeeper, identifying if an image is "out-of-distribution" (OOD). If the image is familiar to the model, only then should aleatoric components be used to visualize potential variations in the segmentation for clinical decision support.

Conclusion

This study highlights the necessity of analyzing the "type of variation" in a dataset before selecting a model architecture. While complex probabilistic models are powerful, they are not always necessary for simple noise. Future work should focus on developing dual-uncertainty frameworks that effectively separate OOD detection from in-distribution variation. By adopting more granular metrics like $D^2_{Det}$ and $D^2_{IoU}$, the medical imaging community can better evaluate and deploy models that reflect the true complexities of clinical expert disagreement.

Summary updated

Abstract

We address the selection and evaluation of uncertain segmentation methods in medical imaging and present two case studies: prostate segmentation, illustrating that for minimal annotator variation simple deterministic models can suffice, and lung lesion segmentation, highlighting the limitations of the Generalized Energy Distance (GED) in model selection. Our findings lead to guidelines for accurately choosing and developing uncertain segmentation models, that integrate aleatoric and epistemic components. These guidelines are designed to aid researchers and practitioners in better developing, selecting, and evaluating uncertain segmentation methods, thereby facilitating enhanced adoption and effective application of segmentation uncertainty in practice.
BibTeX
@inproceedings{zepf2024navigating,
  author = {Kilian Zepf and Jes Frellsen and Aasa Feragen},
  title = {{Navigating Uncertainty in Medical Image Segmentation}},
  booktitle = {ISBI},
  year = {2024},
  doi = {10.1109/isbi56570.2024.10635367},
}
Figure