Distribution-Aware Replay for Continual MRI Segmentation

Nick Lemke, Camila González, Anirban Mukhopadhyay, Martin Mundt

Abstract

Deep learning models in medical imaging frequently encounter performance degradation due to distribution shifts across different scanners and patient demographics. Continual Learning (CL) offers a solution, yet it is often hampered by privacy regulations that prevent data rehearsal and the risk of "silent failures" on unfamiliar data. This paper introduces ccVAE, a distribution-aware replay strategy that leverages a Conditional Variational Autoencoder to learn and replay internal features of a segmentation network. By conditioning on both task identity and slice index, ccVAE achieves state-of-the-art stability in MRI segmentation while providing a built-in mechanism for Out-of-Distribution (OoD) detection.

Introduction

Medical imaging models face a dynamic environment where data distributions shift due to changes in scanner hardware, acquisition protocols, or patient populations. While Continual Learning (CL) aims to update models without forgetting previous knowledge, it faces two primary hurdles in clinical settings. First, privacy regulations like HIPAA and GDPR often prohibit the long-term storage of raw patient data required for traditional rehearsal-based CL. Second, standard segmentation architectures, such as the UNet, tend to be overconfident, producing incorrect segmentations on Out-of-Distribution (OoD) data without warning.

Methods: The ccVAE Architecture

The proposed method augments a standard UNet with a Conditional Variational Autoencoder (ccVAE) to facilitate privacy-preserving pseudo-rehearsal and integrated OoD detection.

The proposed four-stage architecture: (1) Initial UNet training, (2) VAE training on features, (3) Pseudo-replay for subsequent tasks, and (4) Integrated OoD detection during inference.

Feature Replay and Pseudo-rehearsal

Instead of storing raw images, the ccVAE learns the distribution of internal features from the UNet's bottleneck. When the model encounters a new task, the VAE generates synthetic features representing past distributions. These "pseudo-samples" are passed to the UNet's decoder, allowing it to maintain performance on old tasks without access to the original images.

Double Conditioning

To account for the spatial variance in 3D MRI volumes processed as 2D slices, the VAE is doubly conditioned on both the task identity and the slice index. This ensures that the generated features are representative of the specific anatomical context of each slice, which is critical for accurate reconstruction and replay.

Integrated Out-of-Distribution Detection

During inference, the VAE attempts to reconstruct the features of the input image. A high reconstruction error indicates that the input deviates significantly from the training distribution. This allows the system to flag potential failures as OoD rather than providing unreliable segmentations.

Experimental Evaluation

The framework was evaluated on two anatomical targets across multiple institutional datasets:

  • Hippocampus: Utilizing DecathHip, Dryad, and HarP datasets to simulate T1-weighted MRI shifts.
  • Prostate: Using five datasets (BIDMC, I2VCB, HK, UCL, and RUNMC) with varying scanner types and the use of endorectal coils.
Representative MRI slices and ground truth masks from the hippocampus and prostate datasets, illustrating the variability across different slice indices and institutions.

To further test robustness, the researchers introduced synthetic artifacts common in MRI, such as bias fields, spiking, and ghosting, to evaluate the OoD detection capabilities.

Results and Discussion

The ccVAE approach demonstrated significant improvements in both segmentation accuracy and model stability. On the Hippocampus task, the method achieved a Mean Dice score of 87.8%, vastly outperforming sequential training (20.1%) and surpassing other privacy-preserving CL methods like Elastic Weight Consolidation (EWC, 77.5%).

Key insights include:

  • Stability: Unlike competing methods that exhibit volatile performance as new tasks are added, ccVAE maintained a consistent performance trajectory.
  • Reliable Failure Detection: The reconstruction-based OoD detection proved more effective than standard confidence-based metrics, successfully identifying most artifact-laden or domain-shifted images.
  • Privacy Compliance: By replaying features rather than pixels, the method provides a viable path for CL in privacy-sensitive medical environments.

Limitations and Future Work

A primary limitation is the use of a fixed encoder after the first task to maintain a stable feature distribution for the VAE, which may restrict the model's ability to adapt to radically new low-level visual features. Future work will explore dynamic encoder updates and the application of this framework to other medical imaging modalities.

Summary updated

BibTeX
@inproceedings{lemke2024distribution,
  author = {Nick Lemke and Camila González and Anirban Mukhopadhyay and Martin Mundt},
  title = {{Distribution-Aware Replay for Continual MRI Segmentation}},
  booktitle = {AIPAD/PILM @ MICCAI},
  year = {2024},
  url = {https://openreview.net/forum?id=G70pjJVjqS},
}

Loading related content...

Figure