Anatomy of Domain Shift Impact on U-Net Layers in MRI Segmentation

Ivan Zakazov, Boris Shirokikh, Alexey Chernyavskiy, Mikhail Belyaev

In the world of medical imaging, a model trained on data from one hospital often fails when deployed at another. This "domain shift"—caused by different MRI scanners, field strengths, or imaging protocols—is the central headache of medical AI. The standard fix is supervised domain adaptation (sDA): taking a pre-trained model and fine-tuning it on a small amount of labeled data from the new site.

But here’s the catch: Which layers should you actually fine-tune? Should you update the early layers that handle low-level textures, or the deep layers that understand complex shapes? A new paper, "Anatomy of Domain Shift Impact on U-Net Layers in MRI Segmentation," introduces SpotTUnet to solve this by letting the network decide for itself.

TL;DR

SpotTUnet is an adaptive architecture that automatically identifies which specific layers of a U-Net need fine-tuning when moving to a new MRI domain. By learning a "fine-tuning policy," it matches the performance of the best manual strategies while providing a visual "map" of exactly where domain shift hits the network hardest.

What Problem Does This Paper Address?

When adapting a Convolutional Neural Network (CNN) to a new domain, researchers usually pick a heuristic: "fine-tune the whole thing," "only the last layer," or "only the first block."

In computer vision, the conventional wisdom is to tune the last layers (high-level features). However, in medical imaging, some argue that the first layers are more critical because domain shift often manifests as changes in image intensity and noise (low-level features). There is no consensus, and the "optimal" layers to tune likely change depending on the specific scanner or task. This paper aims to remove the guesswork and provide an interpretable way to see which parts of a U-Net are most sensitive to domain shift.

Methods

The authors propose SpotTUnet, an extension of the "SpotTune" concept specifically adapted for the U-Net architecture used in medical segmentation.

  1. The "Twin" Architecture: For every block in the segmentation network, SpotTUnet maintains two versions: a frozen copy (with weights from the source domain) and a trainable copy (to be fine-tuned on the target domain).
  2. The Policy Network: A separate, lightweight ResNet acts as a "manager." For a given input image, it outputs a binary decision for each block: "Use the frozen version" or "Use the fine-tuned version."
  3. Differentiable Decisions: To train this "manager" using standard backpropagation, they use the Gumbel-Softmax trick, which allows gradients to flow through discrete binary choices.
  4. Smart Regularization: The authors introduced a simplified $L_1$ regularization term. This encourages the model to reuse frozen blocks whenever possible, which is crucial for preventing overfitting when you only have a handful of labeled images from the new domain.

Datasets

The study utilizes the CC359 (Calgary-Campinas-359) dataset.

  • Content: 359 high-resolution head MRIs.
  • Task: Skull stripping (segmenting the brain from the skull).
  • Domain Shift: The data is split into 6 distinct domains based on scanner manufacturers (Siemens, GE, Philips) and magnetic field strengths (1.5T, 3T). This provides a rigorous benchmark for testing how models jump from one hardware setup to another.

Key Insights and Ideas

  • The "Domain Shift Stamp": SpotTUnet doesn't just segment images; it provides a visualization of the "anatomy" of the shift. By looking at which layers the policy network chooses to fine-tune, researchers can see exactly where the source model's knowledge breaks down.
  • Encoder Vulnerability: Contrary to some computer vision trends, the results showed that the encoder part of the U-Net (the part that contracts and extracts features) is generally more susceptible to domain shift in MRI than the decoder.
  • Regularization is a Lifesaver: In scenarios with extreme data scarcity (e.g., only 8 or 12 slices of the target domain), the $L_1$ regularization was vital. It forced the model to be "picky" about what it tuned, preventing it from "forgetting" general features and overfitting to the tiny target sample.
  • Histogram Matching Fails: Interestingly, the authors found that traditional intensity-based preprocessing (histogram matching) actually performed worse than the baseline, highlighting the need for more sophisticated neural adaptation.

Key Results

  • Performance Parity: SpotTUnet consistently performed as well as—or better than—the best "hand-picked" manual fine-tuning strategies across the entire spectrum of data availability (from 5 slices to 800).
  • Robustness: While manual strategies (like "Fine-tune First Layers") might work well for one scanner but fail for another, SpotTUnet’s automated approach remained stable across all 30 source-target domain pairs tested.
  • Interpretability: The policy maps (Fig. 4 in the paper) clearly show that as more data becomes available, the model "unlocks" more layers for fine-tuning, starting primarily with the encoder blocks.

Limitations and Future Work

The current version of SpotTUnet is supervised, meaning it still requires some labeled data from the target domain. The authors suggest that the "policy maps" generated by SpotTUnet could be used to design better unsupervised domain adaptation (UDA) methods. For instance, if we know the encoder is the most sensitive part, we could specifically target adversarial training or feature alignment on those specific layers to make models more robust without needing any new labels.

Summary updated

Abstract

Domain Adaptation (DA) methods are widely used in medical image segmentation tasks to tackle the problem of differently distributed train (source) and test (target) data. We consider the supervised DA task with a limited number of annotated samples from the target domain. It corresponds to one of the most relevant clinical setups: building a sufficiently accurate model on the minimum possible amount of annotated data. Existing methods mostly fine-tune specific layers of the pretrained Convolutional Neural Network (CNN). However, there is no consensus on which layers are better to fine-tune, e.g. the first layers for images with low-level domain shift or the deeper layers for images with high-level domain shift. To this end, we propose SpotTUnet – a CNN architecture that automatically chooses the layers which should be optimally fine-tuned. More specifically, on the target domain, our method additionally learns the policy that indicates whether a specific layer should be fine-tuned or reused from the pretrained network. We show that our method performs at the same level as the best of the non-flexible fine-tuning methods even under the extreme scarcity of annotated data. Secondly, we show that SpotTUnet policy provides a layer-wise visualization of the domain shift impact on the network, which could be further used to develop robust domain generalization methods. In order to extensively evaluate SpotTUnet performance, we use a publicly available dataset of brain MR images (CC359), characterized by explicit domain shift. We release a reproducible experimental pipeline ( https://github.com/neuro-ml/domain_shift_anatomy ).
BibTeX
@inproceedings{zakazov2021anatomy,
  author = {Ivan Zakazov and Boris Shirokikh and Alexey Chernyavskiy and Mikhail Belyaev},
  title = {{Anatomy of Domain Shift Impact on U-Net Layers in MRI Segmentation}},
  booktitle = {International Conference on Medical Image Computing and Computer-Assisted Intervention},
  year = {2021},
  url = {https://openreview.net/forum?id=8OjCI9C5sc},
}

Loading related content...

Figure