A Novel Domain Adaptation Framework for Medical Image Segmentation

Amir Gholami, Shashank Subramanian, Varun Shenoy, Naveen Himthani, Xiangyu Yue, Sicheng Zhao, Peter H. Jin, George Biros, Kurt Keutzer

TL;DR

This paper introduces a medical image segmentation framework that tackles data scarcity by blending biophysical tumor growth simulations with CycleGAN to generate realistic training data. It also automatically labels healthy brain structures using image registration, significantly boosting tumor segmentation accuracy on the BraTS'18 challenge dataset.

What Problem Does This Paper Address?

Deep learning models excel at semantic segmentation, but they are notoriously data-hungry. While computer vision datasets like ImageNet boast millions of images, medical imaging datasets are often tiny due to privacy concerns and the immense effort required for expert annotation. For instance, the Brain Tumor Segmentation (BraTS) dataset contains fewer than 300 cases.

Furthermore, radiologists don't just look at a tumor in isolation; they evaluate it relative to surrounding healthy anatomy. However, manually annotating healthy brain structures (like white matter, gray matter, and cerebrospinal fluid) across hundreds of 3D MRI scans is too time-consuming to be practical. This leaves networks to train on highly imbalanced data with minimal anatomical context.

Methods

The authors introduce two core innovations to overcome these data limitations:

  1. Biophysics-Based Domain Adaptation (Data Augmentation): Instead of relying on a standard Generative Adversarial Network (GAN) to create synthetic data from scratch—which can lead to anatomically impossible structures—the authors use a partial differential equation (PDE)-based biophysical tumor model. This model simulates realistic tumor cell diffusion and the physical "mass effect" (how a tumor deforms surrounding brain tissue). Because pure simulations lack realistic MRI intensity distributions, they pass the simulated images through a CycleGAN to bridge the "simulation-to-reality" gap, creating realistic-looking MRIs with perfectly known ground-truth labels.

  2. Extended Segmentation via Image Registration: To provide the network with healthy tissue context, the authors automatically label healthy anatomy in the training data. They perform affine and diffeomorphic image registration to map pre-labeled healthy brain atlases onto the patient scans. A weighted majority-voting scheme fuses these mappings, providing labels for gray matter, white matter, and cerebrospinal fluid without requiring manual radiologist labor.

The Network Pipeline

The overall framework is network-agnostic, but the authors demonstrate it using a two-stage routine:

  • Stage 1: A 10-layer 3D U-Net localizes the general tumor region.
  • Stage 2: A 2D multiview U-Net ensemble (processing axial, sagittal, and coronal slices) takes cropped regions from Stage 1 alongside the augmented data to produce the final fine-grained segmentation.

Datasets

  • BraTS '18 Challenge Dataset: Consists of multimodal 3D MRIs (T1, post-Gadolinium T1ce, T2, and FLAIR) of glioma patients. The training set contains 285 instances. Evaluation was performed on the BraTS'18 validation and testing portals.

Key Insights and Ideas

  • Physics Meets AI: Purely data-driven generative models (like standard GANs) often hallucinate artifacts. By anchoring data generation in a biophysical PDE model, the authors ensure that the underlying anatomy and tumor progression dynamics are biologically plausible before applying style transfer.
  • Healthy Context Matters: Training a network to recognize healthy tissue boundaries actually helps it better delineate tumorous boundaries. It forces the network to learn structural relationships and mitigates the severe class imbalance problem typical in lesion segmentation.

Key Results

The proposed framework yielded substantial performance gains over the baseline network on the BraTS'18 validation set, measured via the Dice score (where higher is better):

  • Enhancing Tumor (ET): 79.15% vs. 73.86% (Baseline)
  • Whole Tumor (WT): 90.81% vs. 89.49% (Baseline)
  • Tumor Core (TC): 81.91% vs. 79.94% (Baseline)

Additionally, this biophysics-grounded domain adaptation approach outperformed existing state-of-the-art pure GAN data augmentation methods by 5 percentage points, even though the authors used a simpler 2D network architecture for their final stage.

Limitations and Future Work

The primary limitation of the current framework is that the domain adaptation step only supports 2D transformations. Consequently, the 3D MRI volumes must be processed slice-by-slice, forcing the authors to rely on a sub-optimal 2D neural network ensemble for the final segmentation stage rather than a fully volumetric 3D pipeline.

The authors note that extending their biophysical domain adaptation framework to full 3D space is the immediate focus of their future work, which they expect will yield even greater improvements in segmentation robustness.

Summary updated

Abstract

We propose a segmentation framework that uses deep neural networks and introduce two innovations. First, we describe a biophysics-based domain adaptation method. Second, we propose an automatic method to segment white and gray matter, and cerebrospinal fluid, in addition to tumorous tissue. Regarding our first innovation, we use a domain adaptation framework that combines a novel multispecies biophysical tumor growth model with a generative adversarial model to create realistic looking synthetic multimodal MR images with known segmentation. Regarding our second innovation, we propose an automatic approach to enrich available segmentation data by computing the segmentation for healthy tissues. This segmentation, which is done using diffeomorphic image registration between the BraTS training data and a set of prelabeled atlases, provides more information for training and reduces the class imbalance problem. Our overall approach is not specific to any particular neural network and can be used in conjunction with existing solutions. We demonstrate the performance improvement using a 2D U-Net for the BraTS'18 segmentation challenge. Our biophysics based domain adaptation achieves better results, as compared to the existing state-of-the-art GAN model used to create synthetic data for training.
BibTeX
@inproceedings{gholami2018novel,
  author = {Amir Gholami and Shashank Subramanian and Varun Shenoy and Naveen Himthani and Xiangyu Yue and Sicheng Zhao and Peter H. Jin and George Biros and Kurt Keutzer},
  title = {{A Novel Domain Adaptation Framework for Medical Image Segmentation}},
  booktitle = {BrainLes @ MICCAI},
  year = {2018},
  doi = {10.1007/978-3-030-11726-9\_26},
}
Figure