Transfer Learning from Partial Annotations for Whole Brain Segmentation

Chengliang Dai, Yuanhan Mo, Elsa D. Angelini, Yike Guo, Wenjia Bai

Abstract

Accurate whole-brain segmentation is a cornerstone of neuroimaging research but is hindered by the scarcity of expert-annotated 3D datasets. We present MO-Net, a multi-task learning framework designed to leverage large-scale, automatically generated partial annotations to improve segmentation performance on small, fully annotated datasets. By pre-training on 4,000 UK Biobank scans with partial labels and fine-tuning with a dual-decoder architecture, MO-Net achieves a Dice score of 0.785 on the MALC dataset, a 26% improvement over training from scratch. This approach significantly reduces the manual labeling burden while maintaining high accuracy across 138 brain structures.

Introduction

Identifying anatomical structures in brain MRI, such as the hippocampus or thalamus, is vital for understanding neurological progression and aging. While traditional registration-based tools (e.g., FSL or SPM) are reliable, their high computational cost and slow processing times limit their utility in large-scale studies. Deep learning offers a faster alternative but typically requires massive amounts of "gold standard" manual annotations, which can take an expert several hours per scan. This paper addresses this data bottleneck by exploring how transfer learning from automatically generated "partial" annotations can empower models trained on very small manual datasets.

Methods

The authors propose a two-stage training scheme centered on a modified 3D U-Net architecture called the Multi-Output Network (MO-Net).

Two-Stage Training Pipeline

  1. Stage 1: Pre-training with Partial Annotations. The model is initially trained on 4,000 MRI scans from the UK Biobank. Instead of expert labels, it uses "partial annotations" (15 sub-cortical structures) generated automatically by the FSL tool. This allows the network to learn robust 3D brain features from a massive dataset without any human labeling effort.
  2. Stage 2: Joint Training (MO-Net). The pre-trained model is converted into a multi-task architecture with a shared encoder and two separate decoders. Decoder A continues to predict the 15 partial structures, while Decoder B learns to predict the full 138 structures from a small set of expert-annotated scans.
Overview of the MO-Net pipeline: pre-training on large-scale automated labels followed by joint training on expert-annotated full brain scans.

Datasets

  • UK Biobank (UKBB): 4,000 scans with automated FSL labels (15 regions) for pre-training.
  • MICCAI 2012 (MALC): 30 subjects with manual annotations for 138 regions, used as the primary benchmark for whole-brain performance.
  • Hammers Adult Atlases (HAA): 20 subjects with manual annotations for 67 regions.

Results and Discussion

MO-Net demonstrates significant performance gains by leveraging partial annotations as a form of multi-task regularization.

Quantitative Performance

On the MALC dataset, MO-Net achieved a mean Dice score of 0.785, substantially outperforming a standard 3D U-Net trained from scratch (0.623). This represents a roughly 26% improvement in overlap accuracy. MO-Net also proved competitive with established methods like SLANT8 and SLANT27, which utilize even larger pre-training sets.

Dice score comparison across key brain structures, highlighting the performance advantage of MO-Net (MON) over fine-tuned and from-scratch U-Net baselines.

Key Insights

  • "Silver Standard" Utility: Noisy, automatically generated labels provide a "warm start" that is far superior to random initialization, allowing the model to learn spatial priors before seeing expert data.
  • Multi-task Regularization: Training on the "easy" task (partial segmentation) alongside the "hard" task (full segmentation) prevents the model from overfitting to the small manual dataset.
  • Generalization Beyond the Teacher: MO-Net often produced more accurate sub-cortical segmentations than the automated FSL tool used to train it, suggesting the network learns to generalize past the errors of its initial "teacher."

Conclusion

MO-Net provides a lightweight yet powerful framework for whole-brain segmentation that drastically reduces the need for manual labeling. By effectively transferring knowledge from large-scale partial annotations, it bridges the gap between traditional automated tools and modern deep learning. Future work will explore deeper architectures and the adaptation of these features to clinical domains involving brain lesions or severe atrophy.

Summary updated

BibTeX
@inproceedings{dai2019transfer,
  author = {Chengliang Dai and Yuanhan Mo and Elsa D. Angelini and Yike Guo and Wenjia Bai},
  title = {{Transfer Learning from Partial Annotations for Whole Brain Segmentation}},
  booktitle = {DART/MIL3ID @ MICCAI},
  year = {2019},
  doi = {10.1007/978-3-030-33391-1\_23},
}
Figure