Abstract We address the selection and evaluation of uncertain segmentation methods in medical imaging and present two case studies: prostate segmentation, illustrating that for minimal annotator variation simple deterministic models can suffice, and lung lesion segmentation, highlighting the limitations of the Generalized Energy Distance (GED) in model selection. Our findings lead to guidelines for accurately choosing and developing uncertain segmentation models, that integrate aleatoric and epistemic components. These guidelines are designed to aid researchers and practitioners in better developing, selecting, and evaluating uncertain segmentation methods, thereby facilitating enhanced adoption and effective application of segmentation uncertainty in practice.
Introduction
In medical artificial intelligence, "uncertainty" is frequently employed as a catch-all term, yet not all uncertainty is created equal. To build reliable clinical tools, researchers must distinguish between aleatoric uncertainty—the inherent randomness in data, such as differing expert opinions on a tumor’s edge—and epistemic uncertainty, which stems from the model’s lack of knowledge regarding unfamiliar data.
This research provides a critical reality check for uncertainty quantification (UQ) in medical image segmentation. The authors identify two primary pitfalls in current practices: the over-engineering of models for simple noise and the reliance on flawed evaluation metrics that conflate different types of error. By analyzing these issues, the study offers a roadmap for more effective model selection and evaluation.
The Problem: Over-Engineering and Flawed Evaluation
Current research often falls into two traps. First, there is a tendency toward over-engineering, where heavy probabilistic machinery is used to model simple "drawing noise" (unsystematic variation like shaky hands) that a standard deterministic model could handle. Second, the field relies heavily on the Generalized Energy Distance (GED) to rank models. GED is problematic because it conflates the difficulty of detecting a lesion with the difficulty of delineating its boundaries. This lack of granularity can lead to misleading conclusions about which model architecture is truly superior for a specific clinical task.
Methodology
The authors compared several popular uncertainty-aware architectures against a standard deterministic U-Net:
- Ensembles: Training multiple models to measure disagreement.
- Probabilistic U-Net: Utilizing latent variables to capture multiple possible segmentation outputs.
- Stochastic Segmentation Networks (SSN): Modeling logit distributions to capture spatial correlations.
- MC-Dropout: A common technique for estimating epistemic uncertainty through stochastic passes during inference.
To address the limitations of GED, the authors proposed decomposing it into two task-specific metrics:
- $D^2_{Det}$ (Detection GED): Measures how well the model captures the ambiguity of whether an object exists at all.
- $D^2_{IoU}$ (Segmentation GED): Measures variation in boundary shapes, specifically excluding cases where the object was missed entirely.
Experimental Design
The study utilized two distinct benchmarks to illustrate different types of uncertainty:
- QUBIQ 2021 (Prostate): A dataset featuring six annotators. Variation here is minimal, consisting mostly of unsystematic drawing errors.
- LIDC-IDRI (Lung Lesions): A highly ambiguous dataset where four radiologists often disagree on whether a lesion exists, as well as its exact shape.
Results and Discussion
The "Softmax is Enough" Revelation
In the prostate segmentation task, where expert agreement is high, the entropy from a simple U-Net’s softmax output provided uncertainty estimates that were as effective as complex aleatoric models. In these instances, adding complexity increases computational overhead without providing additional clinical value. The deterministic U-Net and SSN performed on par, while Ensembles and Probabilistic U-Nets tended to overestimate the variation in the data.
Disentangling Detection and Segmentation
In the lung lesion task, no single model emerged as a universal winner. Ensembles proved best at capturing detection ambiguity ($D^2_{Det}$), but other models were superior at boundary delineation ($D^2_{IoU}$). This confirms that aggregate scores like the original GED can mask a model's true strengths and weaknesses. If a model is excellent at finding lesions but poor at defining their edges, a single GED score might rank it similarly to a model with the opposite profile.
The Epistemic Gatekeeper
The authors propose a "two-step" framework for UQ. Epistemic uncertainty should serve as a gatekeeper, identifying if an image is "out-of-distribution" (OOD). If the image is familiar to the model, only then should aleatoric components be used to visualize potential variations in the segmentation for clinical decision support.
Conclusion
This study highlights the necessity of analyzing the "type of variation" in a dataset before selecting a model architecture. While complex probabilistic models are powerful, they are not always necessary for simple noise. Future work should focus on developing dual-uncertainty frameworks that effectively separate OOD detection from in-distribution variation. By adopting more granular metrics like $D^2_{Det}$ and $D^2_{IoU}$, the medical imaging community can better evaluate and deploy models that reflect the true complexities of clinical expert disagreement.
Summary updated