MS-Net: Multi-Site Network for Improving Prostate Segmentation With Heterogeneous MRI Data

Quande Liu, Qi Dou, Lequan Yu, Pheng-Ann Heng

TL;DR

Deep learning models for medical imaging crave large datasets, but combining MRI scans from different hospitals often degrades performance due to variations in scanners and imaging protocols. This paper introduces MS-Net, a multi-site learning framework that uses Domain-Specific Batch Normalization and a novel Multi-site-guided Knowledge Transfer approach to learn robust, shared representations. By effectively handling data heterogeneity, MS-Net significantly improves prostate segmentation accuracy across multiple clinical sites simultaneously.

What Problem Does This Paper Address?

Training accurate Convolutional Neural Networks (CNNs) for medical image segmentation requires large amounts of annotated data, which is notoriously scarce at any single hospital or clinical site. A logical solution is to pool data from multiple sites.

However, multi-site MRI data is highly heterogeneous. Different hospitals use different scanner manufacturers (e.g., Siemens vs. Philips), magnetic field strengths, coil types, and imaging protocols. This creates a "domain shift." Previous studies have shown that directly mixing this heterogeneous data to train a single network (a "Joint" approach) often yields marginal improvements or even performs worse on local data than a model trained exclusively on that local data (a "Separate" approach). The authors tackle the challenge of effectively aggregating multi-site data to boost performance across all participating sites without falling victim to inter-site heterogeneity.

Methods

The authors propose the Multi-Site Network (MS-Net), which relies on two core innovations to disentangle site-specific noise from shared anatomical features:

  1. Domain-Specific Batch Normalization (DSBN): Instead of sharing Batch Normalization (BN) layers across all data, the network's backbone assigns a dedicated BN layer to each specific site. Because BN layers track the mean and variance of feature maps, forcing all sites through one BN layer muddles the statistics. DSBN allows the network to normalize features individually for each site, absorbing domain-specific nuances so the shared convolutional filters can focus on generic prostate features.
  2. Multi-site-guided Knowledge Transfer (MSKT): To prevent the shared convolutional kernels from just memorizing site-specific quirks, the authors introduce a novel training paradigm. Alongside the main "universal network," they train several "auxiliary branches" (one for each site). These auxiliary branches act as site-specific experts. During training, a knowledge transfer loss forces the universal network to mimic the probability maps of these auxiliary branches. This acts as a powerful regularizer, pushing the universal network to find a robust global minimum ("wide valleys") that works well across all datasets.

Datasets

The authors evaluated their method using T2-weighted prostate MRI scans from three highly heterogeneous sites:

  • Site A: 30 samples from Radboud University Nijmegen Medical Centre (from the NCI-ISBI 2013 challenge).
  • Site B: 30 samples from Boston Medical Center (from the NCI-ISBI 2013 challenge).
  • Site C: 19 samples from Hospital Center Regional University of Dijon-Bourgogne (from the I2CVB dataset).
  • External Test Set: 20 samples from the PROMISE12 challenge, used exclusively to test the model's generalization to completely unseen sites.

Key Insights and Ideas

  • Pre-processing isn't a silver bullet: The authors proved that even with careful intensity normalization (like whitening, bias field correction, and noise filtering), simply mixing datasets ("Joint" training) still fails to beat site-specific models on certain sites. The heterogeneity is too deeply embedded in factors like resolution and coil usage to be fixed by pre-processing alone.
  • Batch Norm holds the domain shift: Visualizing the BN statistics of models trained on different sites reveals massive statistical differences. Giving each site its own BN layer (DSBN) is a highly parameter-efficient way to handle domain shift.
  • Knowledge transfer acts as a multi-site regularizer: By forcing the main network to match the predictions of site-specific auxiliary branches, the model is exposed to richer multi-site information than ground-truth labels alone can provide. This increases the posterior entropy of the model, helping it learn truly generic representations.

Key Results

  • Superior Accuracy: MS-Net achieved an overall Dice Coefficient of 91.66% across the three sites, significantly outperforming both the Joint baseline (90.25%) and the Separate baseline (90.56%).
  • Better Boundary Delineation: MS-Net achieved an overall Average Symmetric Distance (ASD) of 0.77mm, compared to 0.87mm for both the Joint and Separate baselines, indicating much tighter boundary segmentation.
  • State-of-the-Art Comparison: MS-Net outperformed existing multi-domain and multi-site adaptation methods, including Parallel-Adapter (90.88% Dice) and USE-Net (90.60% Dice).
  • Ablation Success: Using DSBN alone improved the Joint baseline to 90.91% Dice. Adding the MSKT training paradigm pushed it to the final 91.66%, proving both components are crucial.

Limitations and Future Work

  • Inference Constraints: Because MS-Net uses Domain-Specific Batch Normalization, it requires knowing which site a test image comes from at inference time so it can route the data through the correct BN layer. (This is usually fine in clinical settings where DICOM headers contain this metadata).
  • Generalization to Unseen Sites: The model is highly optimized for the specific internal sites it was trained on. When tested on completely unseen external sites (the PROMISE12 dataset), it underperforms a standard Jointly-trained model, which naturally learns a more globally spread (if less locally accurate) distribution.
  • Future Work: The authors plan to tackle "Domain Generalization"—developing methods that not only improve performance on known internal sites but also generalize seamlessly to unseen external hospitals. They also plan to apply this multi-site framework to other tasks, such as chest X-ray classification across massive, multi-institutional datasets.

Summary updated

Abstract

Automated prostate segmentation in MRI is highly demanded for computer-assisted diagnosis. Recently, a variety of deep learning methods have achieved remarkable progress in this task, usually relying on large amounts of training data. Due to the nature of scarcity for medical images, it is important to effectively aggregate data from multiple sites for robust model training, to alleviate the insufficiency of single-site samples. However, the prostate MRIs from different sites present heterogeneity due to the differences in scanners and imaging protocols, raising challenges for effective ways of aggregating multi-site data for network training. In this paper, we propose a novel multi-site network (MS-Net) for improving prostate segmentation by learning robust representations, leveraging multiple sources of data. To compensate for the inter-site heterogeneity of different MRI datasets, we develop Domain-Specific Batch Normalization layers in the network backbone, enabling the network to estimate statistics and perform feature normalization for each site separately. Considering the difficulty of capturing the shared knowledge from multiple datasets, a novel learning paradigm, i.e., Multi-site-guided Knowledge Transfer, is proposed to enhance the kernels to extract more generic representations from multi-site data. Extensive experiments on three heterogeneous prostate MRI datasets demonstrate that our MS-Net improves the performance across all datasets consistently, and outperforms state-of-the-art methods for multi-site learning.
BibTeX
@article{liu2020ms,
  author = {Quande Liu and Qi Dou and Lequan Yu and Pheng-Ann Heng},
  title = {{MS-Net: Multi-Site Network for Improving Prostate Segmentation With Heterogeneous MRI Data}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2020},
  url = {https://openreview.net/forum?id=09c8tUscbg},
}

Loading related content...

Figure