First U-Net Layers Contain More Domain Specific Information Than the Last Ones

Boris Shirokikh, Ivan Zakazov, Alexey Chernyavskiy, Irina Fedulova, Mikhail Belyaev

TL;DR

This paper challenges the conventional wisdom of transfer learning by proving that for MRI segmentation, domain shift is primarily a low-level feature problem. The authors demonstrate that fine-tuning the first layers of a U-Net is far more effective than fine-tuning the last layers, and it even outperforms fine-tuning the entire network when target domain data is extremely scarce.

What Problem Does This Paper Address?

Convolutional Neural Networks (CNNs) are the gold standard for medical image segmentation, but they suffer from a major vulnerability: domain shift. If you train a model on MRI scans from one hospital and test it on scans from another, the performance often falls off a cliff. This happens because different MRI scanners (e.g., Siemens vs. GE) and scanning protocols (e.g., 1.5T vs. 3T magnetic fields) produce wildly different image intensities and contrasts.

Most existing domain adaptation (DA) methods try to fix this by fine-tuning the last layers of the network, assuming that high-level, task-specific features need the most adjustment. The authors of this paper tackle this assumption head-on. They hypothesize that because MRI variability is mostly about low-level image intensity, the first layers actually contain the most domain-specific information and should be the primary target for adaptation.

Methods

The authors set up a supervised domain adaptation experiment to compare three fine-tuning strategies:

  1. Fine-tuning all layers (the whole network).
  2. Fine-tuning the first layers only.
  3. Fine-tuning the last layers only.

They used a 2D U-Net architecture upgraded with residual blocks. To rigorously test how these strategies perform under data scarcity—a common reality in medical imaging—they varied the amount of available target domain training data. They tested scenarios ranging from having 3 full MRI scans down to just a tiny fraction of a single scan (e.g., subsampling 1/48th of the 2D slices from one 3D volume).

Datasets

The experiments were conducted on the publicly available CC359 dataset, which the authors highlight as an excellent benchmark for cross-domain adaptation.

  • It contains 359 head MRIs.
  • The data is split across 6 distinct domains, representing combinations of three scanner vendors (Siemens, Philips, General Electric) and two magnetic field strengths (1.5T and 3T).

Key Insights and Ideas

  • Medical Domain Shift is a Low-Level Problem: Contrary to mainstream transfer learning practices (where early layers are frozen because they are assumed to be "general" edge detectors), MRI domain shift fundamentally alters low-level features. Re-learning the image intensity profile at the start of the network is much more efficient than trying to untangle it at the deep, high-level layers.
  • First-Layer Fine-Tuning Acts as a Regularizer: When you have an extremely small amount of annotated data from a new hospital (e.g., just a few slices), fine-tuning the entire network leads to overfitting. Freezing the rest of the network and only updating the first layers prevents this, yielding the best performance in highly constrained setups.
  • Rethinking Unsupervised DA: Many complex unsupervised domain adaptation methods (like those using Gradient Reversal Layers) focus on aligning deep, bottleneck features. This paper's findings suggest the community might be looking at the wrong end of the network.

Key Results

  • The Baseline Drop: Without any fine-tuning, transferring a model across domains causes massive degradation. For example, the Surface Dice Score (a metric sensitive to boundary delineation) dropped from a healthy 0.85–0.89 down to as low as 0.09 in the worst-case cross-domain transfers.
  • First vs. Last: Across almost all 30 possible source-target domain pairs, fine-tuning the first layers significantly and consistently outperformed fine-tuning the last layers.
  • First vs. All: When target data was severely limited (between 1/48th and 1/12th of a single scan), fine-tuning only the first layers outperformed fine-tuning all layers. As the amount of target data increased to 1–3 full scans, fine-tuning all layers eventually took the lead, though first-layer fine-tuning remained highly competitive.

Limitations and Future Work

The authors acknowledge that their study focuses on a simple supervised setup, meaning it still requires at least a small amount of manually annotated data in the new target domain. Additionally, the core experiments rely on a 2D U-Net, though they note observing similar baseline degradation in 3D U-Nets.

For future work, the authors strongly suggest that unsupervised domain adaptation approaches—which require no labeled target data—could see massive benefits by shifting their focus to target the first layers of the network rather than the last ones.

Summary updated

Abstract

MRI scans appearance significantly depends on scanning protocols and, consequently, the data-collection institution. These variations between clinical sites result in dramatic drops of CNN segmentation quality on unseen domains. Many of the recently proposed MRI domain adaptation methods operate with the last CNN layers to suppress domain shift. At the same time, the core manifestation of MRI variability is a considerable diversity of image intensities. We hypothesize that these differences can be eliminated by modifying the first layers rather than the last ones. To validate this simple idea, we conducted a set of experiments with brain MRI scans from six domains. Our results demonstrate that 1) domain-shift may deteriorate the quality even for a simple brain extraction segmentation task (surface Dice Score drops from 0.85–0.89 even to 0.09); 2) fine-tuning of the first layers significantly outperforms fine-tuning of the last layers in almost all supervised domain adaptation setups. Moreover, fine-tuning of the first layers is a better strategy than fine-tuning of the whole network, if the amount of annotated data from the new domain is strictly limited.
BibTeX
@inproceedings{shirokikh2020first,
  author = {Boris Shirokikh and Ivan Zakazov and Alexey Chernyavskiy and Irina Fedulova and Mikhail Belyaev},
  title = {{First U-Net Layers Contain More Domain Specific Information Than the Last Ones}},
  booktitle = {DART/DCL @ MICCAI},
  year = {2020},
  url = {https://openreview.net/forum?id=u2WWSCrUQ1},
}
Figure