Deep Learning with Mixed Supervision for Brain Tumor Segmentation

Pawel Mlynarski, Hervé Delingette, Antonio Criminisi, Nicholas AyachearXiv 2018 · DOI · arXiv

TL;DR

This paper introduces a "mixed supervision" deep learning approach for brain tumor segmentation that trains a network on a combination of scarce, expensive pixel-level annotated images and abundant, cheap image-level labeled images. By extending a standard U-Net with dedicated image-classification branches, the model leverages weak labels to significantly boost segmentation accuracy when fully annotated data is limited.

What Problem Does This Paper Address?

Deep learning has revolutionized medical image segmentation, but its success heavily relies on large datasets with pixel-level annotations. In medical imaging, creating these "ground truth" masks is a major bottleneck: manual delineation of tumors is tedious, time-consuming (taking up to 45 minutes per MRI), and requires highly specialized medical expertise. Even among experts, subjective boundaries lead to an inter-rater Dice overlap of only 0.74 to 0.85.

Conversely, weak annotations—such as a global binary label indicating whether a tumor is simply present or absent in a scan slice—are much cheaper and faster to acquire. This paper addresses how to effectively combine a small handful of fully annotated scans with a larger pool of weakly annotated scans to train high-performing segmentation models, a paradigm the authors call mixed supervision.

Different levels of supervision compared to the proposed mixed supervision approach.

Methods

The authors propose a novel multi-task convolutional neural network (CNN) architecture that performs both pixel-level segmentation and image-level classification simultaneously.

  1. Architecture Extension: They take a standard 2D U-Net and branch off from its second-to-last convolutional layer (just before the final segmentation output). This branch consists of a mean-pooling layer, a convolutional layer, and seven fully connected layers with a skip connection to output global classification scores.
  2. Joint Training & Loss: The network is optimized using a combined loss function: $$\text{Loss} = a \cdot \text{Loss}_s + (1 - a) \cdot \text{Loss}_c$$ where $\text{Loss}_s$ is a weighted pixelwise cross-entropy loss (applied only to fully annotated images and completely healthy images) and $\text{Loss}_c$ is a standard cross-entropy classification loss (applied to all images).

  3. Smart Batching: To stabilize training, each mini-batch is carefully balanced to contain a fixed ratio of three image types: $k$ positive cases with full segmentation, $m$ negative cases (no tumor), and $n$ positive cases with only image-level labels.

  4. Multiclass Extension: To handle multiple tumor sub-components (edema, non-enhancing core, enhancing core), the architecture is expanded by giving each tumor subclass its own independent, dedicated classification branch.
Figure 2: Architecture of our model for binary segmentation. The numbers of outputs are specified below boxes representing layers. The height of rectangles represents the scale (increasing with pooling operations). The dashed lines represent concatenation operations. The proposed architecture is an extended version of U-net, with a subnetwork performing image-level classification. Training of the model corresponds to a joint minimization of two loss functions related respectively to segmentation and image-level classifcation tasks.

Datasets

The method was evaluated on the BRATS 2018 challenge training dataset, which contains 285 multisequence MRIs (T1, T1+gadolinium, T2, and FLAIR) of patients diagnosed with low-grade or high-grade gliomas. The authors framed the evaluation as a 2D problem using axial slices.

Key Insights and Ideas

  • Feature Sharing via Late Branching: Branching the classification network from the end of the U-Net decoder rather than the encoder ensures that the classification task directly guides and refines the deep, spatially rich representations needed for segmentation. It also improves gradient flow throughout the entire network.
  • Subclass Complexity Demands Dedicated Branches: For multiclass segmentation, a single multi-label classification branch performs poorly. Giving each tumor subclass its own dedicated classification network provides the capacity required to learn distinct, subtle features for complex sub-regions like edema versus necrotic cores.
  • Data Substitution Efficiency: Weakly supervised data acts as an effective multiplier for fully supervised data. The mixed supervision model can achieve equivalent performance to a fully supervised model while requiring significantly fewer manual segmentations.

Key Results

The authors simulated data-scarce scenarios using 5-fold cross-validation, varying the number of Fully Annotated (FA) patients vs. Weakly Annotated (WA) patients:

  • The 5 FA Scenario: With only 5 fully annotated patient scans available, a standard U-Net achieved a Mean Dice score of 70.39% for Whole Tumor segmentation. Adding 223 weakly annotated scans boosted the mixed supervision model's Dice score to 78.34% (an 8-point absolute improvement).
  • Efficiency Gains: The performance of the mixed supervision model trained on just 5 FA + 223 WA scans (78.34% Dice) slightly outperformed a standard fully supervised U-Net trained on 15 FA scans alone (77.91% Dice).
  • Diminishing Returns: As the baseline pool of fully annotated data grows, the benefit of weak supervision shrinks. At 30 FA scans, the standard supervised approach and the mixed supervision approach achieved nearly identical results (~83.9% Dice).

Limitations and Future Work

  • 2D vs. 3D Segmentation: The current model operates on 2D axial slices rather than full 3D volumes. The authors note that a 3D extension would capture richer spatial context but was limited here because the BRATS dataset lacks completely tumor-free 3D volumes to properly train a 3D classification branch.
  • Scale of Weak Supervision: The study only utilized around 220 weakly annotated MRIs. Testing the framework on a truly large-scale dataset containing thousands or millions of weakly annotated images remains an open and promising direction.

Summary updated

Abstract

Most of the current state-of-the-art methods for tumor segmentation are based on machine learning models trained on manually segmented images. This type of training data is particularly costly, as manual delineation of tumors is not only time-consuming but also requires medical expertise. On the other hand, images with a provided global label (indicating presence or absence of a tumor) are less informative but can be obtained at a substantially lower cost. In this paper, we propose to use both types of training data (fully-annotated and weakly-annotated) to train a deep learning model for segmentation. The idea of our approach is to extend segmentation networks with an additional branch performing image-level classification. The model is jointly trained for segmentation and classification tasks in order to exploit information contained in weakly-annotated images while preventing the network to learn features which are irrelevant for the segmentation task. We evaluate our method on the challenging task of brain tumor segmentation in Magnetic Resonance images from BRATS 2018 challenge. We show that the proposed approach provides a significant improvement of segmentation performance compared to the standard supervised learning. The observed improvement is proportional to the ratio between weakly-annotated and fully-annotated images available for training.
BibTeX
@misc{mlynarski2018deep,
  author = {Pawel Mlynarski and Hervé Delingette and Antonio Criminisi and Nicholas Ayache},
  title = {{Deep Learning with Mixed Supervision for Brain Tumor Segmentation}},
  year = {2018},
  eprint = {1812.04571},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.04571},
}

Loading related content...

Figure