Reviving Poor Object Segmentations in OOD Medical Images using Variational-Deep-PCA Modeling on Segmentation Maps with Sampling-Free Learning

Jimut B. Pal, Shantanu Welling, Himali Saini, Suyash P. AwateWACV 2025

TL;DR

This paper introduces VarDeepPCA, a novel framework that models the inherent variability in medical image segmentation maps. By learning principal modes of variation, VarDeepPCA can effectively "revive" poor segmentations produced by standard deep learning models on out-of-distribution (OOD) images, significantly improving accuracy with minimal human intervention.

What Problem Does This Paper Address?

Deep neural networks (DNNs) for medical image segmentation often struggle when presented with out-of-distribution (OOD) images. These OOD images can arise from variations in imaging equipment, protocols, or patient populations, leading to segmentations that are inaccurate or contain gross errors. While DNNs are powerful, they typically fail to generalize to data significantly different from their training set. The authors observe that the underlying geometric variations in segmentation maps themselves are often much smaller than the variations in the raw images. This suggests an opportunity to model the variability of segmentation maps directly, rather than relying solely on image-based models that are sensitive to OOD shifts. The challenge is to do this efficiently and effectively, especially when the OOD segmentations are extremely poor.

Methods

The core of the proposed method is the VarDeepPCA framework. It consists of two main components:

  1. Variational-Deep-PCA Model on Segmentation Maps: This component learns the principal modes of variation within a class of segmentation maps. It uses a variational encoder-decoder architecture.
  2. Encoder: Maps an input segmentation map to a latent representation on a simplex. This representation captures the mixture proportions of different modes of variation.
  3. Decoder: Reconstructs a segmentation map from the latent representation.
  4. Sampling-Free Variational Learning: A key innovation is the use of a Bayesian interpretation of the softmax function. This allows for closed-form analytical solutions for variational inference, eliminating the need for expensive Monte Carlo sampling, which is common in methods like Variational Autoencoders (VAEs). This also enables efficient uncertainty estimation.
  5. Reviving Poor Segmentations:
    • Automatic Revival: For moderately poor segmentations, the framework first filters the input segmentation map using the learned encoder-decoder. Then, it "projects" this filtered map onto the learned principal modes of variation by optimizing a latent representation via gradient descent.
  6. Human-in-the-Loop Revival: For extremely poor segmentations, especially for objects with star-convex geometry, a human annotator provides very sparse annotations (e.g., drawing arcs for inner/outer boundaries in a few sectors). VarDeepPCA then uses these sparse annotations and its learned model to reconstruct a high-quality segmentation. This approach significantly reduces human effort (e.g., requiring only 5.6% of the annotation effort).

Datasets

The paper evaluates its approach on several publicly available datasets:

  • Myocardium (Heart) Datasets:
    • CAP (used for training and in-distribution testing)
    • ACDC (OOD testing)
    • A-CMRI (OOD testing)
  • Retina Datasets:
    • Magrabi (used for training and in-distribution testing)
    • ORIGA (OOD testing)
    • G1020 (OOD testing)

The authors mention using a small in-distribution training set (150 images) and a validation set (50 images) for hyperparameter tuning.

Key Insights and Ideas

  • Segmentation Map Variability is Lower: The core insight is that the geometric variability of segmentation maps is often much lower than that of the raw medical images, making segmentation maps a more robust target for modeling OOD robustness.
  • Sampling-Free Variational Learning: By reinterpreting the softmax function through a Bayesian lens, the authors achieve sampling-free variational learning and inference. This is a significant computational advantage over traditional VAEs.
  • "Projecting" onto Learned Modes: The idea of "projecting" a poor segmentation onto the learned principal modes of variation is a powerful way to correct errors and enforce anatomical plausibility.
  • Human-in-the-Loop Efficiency: The human-in-the-loop approach demonstrates that even minimal human input, when combined with a strong learned model, can yield excellent results for challenging OOD cases.

Key Results

  • Significant Improvement on OOD Data: VarDeepPCA-augmented versions of existing DNNs consistently and statistically significantly outperform their baseline counterparts on OOD datasets for both myocardium and retina segmentation.
  • Performance Parity on In-Distribution Data: On in-distribution data, VarDeepPCA-augmented models perform comparably to their baselines, with slight variations (slightly worse for myocardium, slightly better for retina).
  • Efficiency: The VarDeepPCA model is lightweight, with the ResUNet+++VarDeepPCA variant having only 6M parameters, significantly outperforming larger models like DSTransUNet (171M parameters).
  • Human-in-the-Loop Effectiveness: The human-in-the-loop method, requiring only 5.6% of annotation effort, achieves highly accurate segmentations on extremely poor OOD cases, outperforming baselines and even the automatic VarDeepPCA revival.
  • Qualitative Improvements: Figures 4, 5, 6, and 7 visually demonstrate the dramatic improvement in segmentation quality, reducing gross errors and implausible shapes on OOD images.

Limitations and Future Work

  • Star-Convex Geometry Assumption: The human-in-the-loop method is specifically noted to work well for objects with star-convex boundaries. This implies a limitation for objects with more complex or non-star-convex geometries.
  • Modality Independence: While the framework is presented as modality-independent, its effectiveness might still vary across different imaging modalities and object types.
  • Future Work: The paper suggests that the learned VarDeepPCA model can be used for various downstream tasks, such as anomaly detection or generating anatomically plausible segmentations. The efficient uncertainty estimation also opens avenues for more robust clinical decision support.

Summary updated

BibTeX
@inproceedings{pal2025reviving,
  author = {Jimut B. Pal and Shantanu Welling and Himali Saini and Suyash P. Awate},
  title = {{Reviving Poor Object Segmentations in OOD Medical Images using Variational-Deep-PCA Modeling on Segmentation Maps with Sampling-Free Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure