Adversarial Continual Learning for Multi-domain Hippocampal Segmentation

Marius Memmel, Camila González, Anirban Mukhopadhyay

Abstract

Deep learning for medical imaging suffers from temporal and privacy-related restrictions on data availability. To still obtain viable models, continual learning aims to train in sequential order, as and when data is available. The main challenge that continual learning methods face is to prevent catastrophic forgetting, i.e., a decrease in performance on the data encountered earlier. This issue makes continuous training of segmentation models for medical applications extremely difficult. Yet, often, data from at least two different domains is available which we can exploit to train the model in a way that it disregards domain-specific information. We propose an architecture that leverages the simultaneous availability of two or more datasets to learn a disentanglement between the content and domain in an adversarial fashion. The domain-invariant content representation then lays the base for continual semantic segmentation. Our approach takes inspiration from domain adaptation and combines it with continual learning for hippocampal segmentation in brain magnetic resonance imaging (MRI). We showcase that our method reduces catastrophic forgetting and outperforms state-of-the-art continual learning methods.

The Challenge of Catastrophic Forgetting

Deep learning models thrive on massive, diverse datasets, but in medical imaging, data is often siloed due to strict privacy regulations and temporal restrictions. To build useful models, researchers use continual learning (CL)—training a model sequentially as new datasets become available over time.

However, CL faces a massive hurdle: catastrophic forgetting. When a model learns from a new batch of data, it tends to overwrite its previous knowledge, causing performance on older datasets to plummet. In medical imaging, this is heavily exacerbated by the fact that different datasets come from different "domains" (e.g., different MRI scanner vendors, varying imaging protocols, or different patient diseases). Existing CL methods try to prevent forgetting by penalizing weight changes or saving old data, but they fail to exploit a common real-world scenario: hospitals usually have at least two datasets available at the start of training (e.g., open-source data plus historical internal data). The authors ask: Can we use this initial multi-domain data to teach the model to ignore scanner-specific noise before we even start sequential training?

The ACS Framework

The authors propose the Adversarial Continual Segmenter (ACS), which combines concepts from image-to-image translation and domain adaptation. Instead of just training a standard segmentation network, ACS splits the input image into two distinct latent representations: a Content Representation ($z_c$), which encodes the actual anatomy needed for segmentation, and a Domain Representation ($z_d$), which captures scanner-specific style.

The ACS architecture uses adversarial discriminators to disentangle anatomical content from domain-specific noise.

To ensure these two streams don't mix, ACS uses adversarial training. A domain discriminator ($D_c$) tries to guess which dataset an image came from based on its content representation. The content encoder ($E_c$) is trained to fool this discriminator, forcing it to strip away all domain-specific information. Crucially, the authors also apply a discriminator to the U-Net's skip connections, which are notorious for leaking domain information past the network's bottleneck.

Continual Learning Strategy

The training process is divided into distinct stages:

  1. Stage 1 (Disentanglement): ACS is trained on at least two datasets simultaneously to learn the content-domain disentanglement.
  2. Stage 2 (Continual Learning): When a new, unseen dataset arrives, the disentanglement encoders are frozen. The model only fine-tunes the last four convolutional layers of the U-Net segmenter ($S$).

Because the segmenter is already operating on a purified, domain-invariant "content" signal, it can learn the new data without catastrophically forgetting the old. This approach posits that forgetting is primarily a domain problem—the model overfitting to the specific noise and contrast of the newest MRI scanner.

Experimental Setup

The method was evaluated on three distinct T1-weighted brain MRI datasets for hippocampus segmentation:

  • Dataset A (Medical Segmentation Decathlon 2018): 195 subjects acquired using Philips scanners.
  • Dataset B (Scientific Data): 25 healthy subjects acquired using various 3 Tesla MRI systems.
  • Dataset C (ADNI): 68 subjects acquired using Siemens, GE, and Philips scanners.

Results and Discussion

ACS demonstrated a significant reduction in forgetting compared to standard U-Nets and state-of-the-art CL methods like BS-MAS (parameter-importance) and OL-KD (knowledge distillation).

Quantitative comparison of ACS against baselines across three dataset combinations.

When trained on the first two datasets (Stage 1), ACS outperformed all baselines, achieving an average Dice score improvement of +0.178. When the third dataset was introduced sequentially, a standard U-Net's performance on Dataset A dropped by over 35% Dice. ACS effectively mitigated this decline, increasing the average Dice score by +0.127 over a standard U-Net and +0.118 over BS-MAS.

Conclusion

ACS shows that disentanglement is a powerful alternative to traditional regularization in continual learning. By isolating a domain-agnostic representation, the model becomes robust to scanner-specific noise. While the method requires at least two diverse datasets to start, it paves the way for the continuous, safe deployment of deep learning models in clinical environments where data arrives sequentially from varying sources.

Summary updated

Abstract

Deep learning for medical imaging suffers from temporal and privacy-related restrictions on data availability. To still obtain viable models, continual learning aims to train in sequential order, as and when data is available. The main challenge that continual learning methods face is to prevent catastrophic forgetting, i.e., a decrease in performance on the data encountered earlier. This issue makes continuous training of segmentation models for medical applications extremely difficult. Yet, often, data from at least two different domains is available which we can exploit to train the model in a way that it disregards domain-specific information. We propose an architecture that leverages the simultaneous availability of two or more datasets to learn a disentanglement between the content and domain in an adversarial fashion. The domain-invariant content representation then lays the base for continual semantic segmentation. Our approach takes inspiration from domain adaptation and combines it with continual learning for hippocampal segmentation in brain magnetic resonance imaging (MRI). We showcase that our method reduces catastrophic forgetting and outperforms state-of-the-art continual learning methods.
BibTeX
@inproceedings{memmel2021adversarial,
  author = {Marius Memmel and Camila González and Anirban Mukhopadhyay},
  title = {{Adversarial Continual Learning for Multi-domain Hippocampal Segmentation}},
  booktitle = {DART/FAIR @ MICCAI},
  year = {2021},
  url = {https://openreview.net/forum?id=Mu3wimrOSo},
}
Figure