PHiSeg: Capturing Uncertainty in Medical Image Segmentation

Christian F. Baumgartner, Kerem Can Tezcan, Krishna Chaitanya, Andreas M. Hötker, Urs J. Muehlematter, Khoschy Schawkat, Anton S. Becker, Olivio Donati, Ender Konukoglu

Abstract

Segmentation of anatomical structures and pathologies is inherently ambiguous. For instance, structure borders may not be clearly visible or different experts may have different styles of annotating. The majority of current state-of-the-art methods do not account for such ambiguities but rather learn a single mapping from image to segmentation. In this work, we propose a novel method to model the conditional probability distribution of the segmentations given an input image. We derive a hierarchical probabilistic model, in which separate latent variables are responsible for modelling the segmentation at different resolutions. Inference in this model can be efficiently performed using the variational autoencoder framework. We show that our proposed method can be used to generate significantly more realistic and diverse segmentation samples compared to recent related work, both, when trained with annotations from a single or multiple annotators.

Introduction

In medical imaging, a single "correct" segmentation often does not exist. Whether delineating a tumor boundary on a CT scan or a prostate gland on an MRI, experts frequently produce varying annotations based on personal interpretation of low-contrast regions. Standard deterministic models, such as the U-Net, learn a one-to-one mapping from image to mask, forcing the network to produce a single "average" result. This approach can lead to over-confident predictions or misdiagnosis in ambiguous clinical cases. While recent probabilistic models have attempted to address this, they often lack sample diversity because stochasticity is typically introduced at only a single scale.

Methods

We propose PHiSeg (Probabilistic Hierarchical Segmentation), a hierarchical conditional variational autoencoder (cVAE) designed to capture multi-scale uncertainty. The architecture moves beyond global random variables by employing a hierarchy of latent variables ($z_1, \dots, z_L$), where each level models uncertainty at a specific resolution.

PHiSeg hierarchical architecture and multi-resolution segmentation process.

Inspired by Laplacian pyramids, PHiSeg generates a segmentation at a low resolution and continuously refines it through increasingly higher resolutions. Each step adds a "residual" refinement governed by the corresponding level's latent variable. This hierarchical structure, integrated into a U-Net-like framework with sampling steps replacing standard skip connections, prevents the network from ignoring the random noise—a common failure mode in simpler cVAEs. To ensure each level learns its assigned resolution, we utilize "deep supervision" with intermediate loss functions at every stage of the hierarchy.

Datasets

The model was evaluated on two challenging tasks:

  1. LIDC-IDRI: A public dataset of 1,018 thoracic CT scans. Each lung lesion was annotated by four radiologists, providing a benchmark for inter-expert variability.
  2. Prostate MR Dataset: An in-house dataset of 68 patients with T2-weighted MRIs, annotated by six experts (four radiologists and two non-radiologists) focusing on the transition and peripheral zones.

Results and Discussion

PHiSeg was compared against the Probabilistic U-Net and deterministic baselines using the Generalized Energy Distance ($D^2_{GED}$), which quantifies the match between the distribution of generated samples and expert annotations.

  • Superior Diversity: PHiSeg ($L=5$) achieved a $D^2_{GED}$ of 0.2248 on LIDC-IDRI, outperforming the Probabilistic U-Net (0.2393). On the Prostate dataset, the improvement was more pronounced (0.0864 vs 0.1322).
  • Uncertainty Calibration: The model produces "$\gamma$-maps" (pixel-wise uncertainty visualizations) that correlate strongly with actual expert disagreement. This indicates the model effectively identifies regions of high ambiguity.
  • Performance Parity: Importantly, the mean prediction of PHiSeg maintains the raw segmentation accuracy (Dice score) of standard deterministic U-Nets, providing uncertainty quantification without a performance penalty.

Conclusion

PHiSeg successfully models the inherent ambiguity of medical images by generating diverse and realistic segmentation hypotheses. By forcing the network to construct masks piece-by-piece through a latent hierarchy, it captures both large-scale structural uncertainty and fine-scale boundary variations. Future work will focus on integrating these multiple hypotheses into clinical workflows, such as radiotherapy planning and surgical margin estimation.

Summary updated

Abstract

Segmentation of anatomical structures and pathologies is inherently ambiguous. For instance, structure borders may not be clearly visible or different experts may have different styles of annotating. The majority of current state-of-the-art methods do not account for such ambiguities but rather learn a single mapping from image to segmentation. In this work, we propose a novel method to model the conditional probability distribution of the segmentations given an input image. We derive a hierarchical probabilistic model, in which separate latent variables are responsible for modelling the segmentation at different resolutions. Inference in this model can be efficiently performed using the variational autoencoder framework. We show that our proposed method can be used to generate significantly more realistic and diverse segmentation samples compared to recent related work, both, when trained with annotations from a single or multiple annotators.
BibTeX
@inproceedings{baumgartner2019phiseg,
  author = {Christian F. Baumgartner and Kerem Can Tezcan and Krishna Chaitanya and Andreas M. Hötker and Urs J. Muehlematter and Khoschy Schawkat and Anton S. Becker and Olivio Donati and Ender Konukoglu},
  title = {{PHiSeg: Capturing Uncertainty in Medical Image Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2019},
  url = {https://openreview.net/forum?id=AEH6WwMU2l},
}
Figure