Continual atlas-based segmentation of prostate MRI

Amin Ranem, Camila González, Daniel Pinto dos Santos, Andreas M. Bucher, Ahmed E. Othman, Anirban Mukhopadhyay

Abstract

Deep learning models in medical imaging often struggle with catastrophic forgetting when encountering new data distributions. This paper introduces Atlas Replay, a continual learning framework that replaces raw data rehearsal with privacy-preserving anatomical prototypes. By framing segmentation as an atlas-based registration task, the method ensures anatomical consistency and robust knowledge retention across diverse prostate MRI datasets.

Introduction: The Challenge of Catastrophic Forgetting

In clinical settings, deep learning models are frequently updated as new data arrives from different hospitals or scanners. However, standard models suffer from catastrophic forgetting, where learning new tasks degrades performance on previously mastered ones. While "rehearsal" (storing old data) is a common solution, strict privacy regulations like GDPR often prohibit the long-term storage of raw patient scans.

The stability-plasticity trade-off and failures in standard regularization-based continual learning.

Furthermore, conventional end-to-end segmentation models (e.g., U-Nets) lack anatomical constraints. When faced with domain shift, they may produce semantically nonsensical results. There is a critical need for a system that balances "plasticity" (learning new domains) with "rigidity" (preserving anatomical knowledge) while respecting patient privacy.

Methods: Atlas Replay and Privacy-Preserving Prototypes

The authors propose Atlas Replay, which revitalizes atlas-based segmentation using modern deep learning.

1. Prototype Building

Instead of storing individual patient scans, the framework creates "prototypes." A small subset of scans ($r=7$) from a specific domain is rigidly aligned and averaged. This process obscures unique patient identifiers while retaining the general structural characteristics of the domain (e.g., specific MRI coil types).

Creation of a privacy-preserving prototype by averaging multiple subjects.

2. Registration-Based Segmentation

During training, the model learns to register (warp) a prototype's pre-defined mask onto a new patient's scan using the VoxelMorph framework. This approach leverages structural domain knowledge as a buffer against forgetting. As new datasets arrive, the model is trained to register the most appropriate prototype, ensuring that the predicted segmentations always maintain a valid anatomical shape.

Experimental Setup and Datasets

The framework was evaluated on seven diverse prostate segmentation datasets: RUNMC, BMC, HCRUDB, UCL, BIDMC, HK, and DecathProst. These datasets encompass various vendors (Siemens, Philips, GE) and acquisition parameters, providing a rigorous test for domain shift and continual learning performance.

Key Results

  • Knowledge Retention: Atlas Replay achieved a Backward Transfer (BWT) of -8.13%, significantly outperforming sequential training (-26.60%) and Elastic Weight Consolidation (EWC, -27.48%).
  • Generalization: The method outperformed standard U-Net models by an average of 15% in Dice score on unseen domains, demonstrating superior robustness to clinical site variations.
  • Privacy Validation: In a user study, senior radiologists could only correctly match a prototype to its source scan with 47.5% precision (near random chance in a 1-of-3 setup), validating the privacy-preserving nature of the averaged prototypes.

Conclusion and Future Directions

Atlas Replay offers a promising path for deploying continual learning in privacy-sensitive medical environments. By shifting from pixel-wise prediction to registration-based transformation, the model maintains high semantic coherence. Future work will focus on automating prototype selection and exploring more rigorous privacy frameworks like Differential Privacy to further enhance clinical utility.

Summary updated

Abstract

Continual learning (CL) methods designed for natural image classification often fail to reach basic quality standards for medical image segmentation. Atlas-based segmentation, a well-established approach in medical imaging, incorporates domain knowledge on the region of interest, leading to semantically coherent predictions. This is especially promising for CL, as it allows us to leverage structural information and strike an optimal balance between model rigidity and plasticity over time. When combined with privacy-preserving prototypes, this process offers the advantages of rehearsal-based CL without compromising patient privacy. We propose Atlas Replay, an atlas-based segmentation approach that uses prototypes to generate high-quality segmentation masks through image registration that maintain consistency even as the training distribution changes. We explore how our proposed method performs compared to state-of-the-art CL methods in terms of knowledge transferability across seven publicly available prostate segmentation datasets. Prostate segmentation plays a vital role in diagnosing prostate cancer, however, it poses challenges due to substantial anatomical variations, benign structural differences in older age groups, and fluctuating acquisition parameters. Our results show that Atlas Replay is both robust and generalizes well to yet-unseen domains while being able to maintain knowledge, unlike end-to-end segmentation methods. Our code base is available under https://github.com/MECLabTUDA/Atlas-Replay.
BibTeX
@inproceedings{ranem2024continual,
  author = {Amin Ranem and Camila González and Daniel Pinto dos Santos and Andreas M. Bucher and Ahmed E. Othman and Anirban Mukhopadhyay},
  title = {{Continual atlas-based segmentation of prostate MRI}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00739},
}
Figure