Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation

Konstantinos Kamnitsas, Wenjia Bai, Enzo Ferrante, Steven G. McDonagh, Matthew Sinclair, Nick Pawlowski, Martin Rajchl, Matthew C. H. Lee, Bernhard Kainz, Daniel Rueckert, Ben Glocker

TL;DR

The EMMA (Ensembles of Multiple Models and Architectures) framework tackles the inherent bias and sensitivity of deep learning models by aggregating predictions from a diverse set of 3D CNN architectures. By "averaging away" the influence of specific meta-parameters and training configurations, the authors created a highly robust system that won first place in the prestigious BRATS 2017 brain tumor segmentation challenge.

What Problem Does This Paper Address?

Deep learning has revolutionized medical image segmentation, but individual models are notoriously finicky. A network's performance is heavily influenced by "meta-parameters"—architectural choices (like receptive field size), loss functions (Dice vs. Cross-Entropy), and preprocessing steps (intensity normalization).

This sensitivity creates two major problems:

  1. Overfitting to Configurations: A model manually tuned to perform perfectly on one dataset might fail on another because its specific configuration is over-fit to the training distribution.
  2. Biased Analysis: If a model is sensitive to specific textures due to its architecture, any clinical findings derived from its segmentations are inherently biased by those architectural choices.

The authors argue that we need "objective" models that are invariant to these arbitrary configuration choices.

Methods

The core contribution is EMMA, which treats meta-parameters as stochastic variables and marginalizes their effect through ensembling. Unlike typical ensembles that use the same model with different random seeds, EMMA combines fundamentally different architectures:

  • DeepMedic: A multi-scale 3D CNN that processes context at different resolutions in parallel.
  • FCN (Fully Convolutional Networks): 3D versions including residual-based variants.
  • U-Net: 3D adaptations using both summation and concatenation for skip connections.

Beyond architectural diversity, the models were trained using different loss functions (Cross-Entropy, IoU, and Dice) and different intensity normalization schemes. At test time, the final segmentation is produced by simply averaging the class-confidence maps from all individual models.

Fig. 2: Our ensemble of diverse networks, EMMA (red), averages out the bias infused by individual model configurations m, to approximate more reliably the true posterior (black), while being robust to suboptimal configurations. Poste- riors on the left were obtained from multiple perceptrons, trained to classify clusters centred on 10 and -10 as a toy example, with different losses, regulariza- tions and noise in the training labels. Their ensemble provides reliable estimates.

Datasets

The system was evaluated using the BRATS 2017 (Brain Tumour Segmentation Challenge) dataset:

  • Training Set: 285 cases (210 High Grade Glioma, 75 Low Grade Glioma) with manual annotations for necrotic core, edema, and enhancing tumor.
  • Validation Set: 46 cases.
  • Test Set: 146 cases (used for the final competition ranking).
  • Modalities: Each case includes four MRI sequences: FLAIR, T1, T1-contrast enhanced (T1ce), and T2.

Key Insights and Ideas

  • Ensembling for Objectiveness: The authors frame ensembling not just as a performance booster, but as a way to achieve "configuration-invariance." By covering a subspace of high-quality models, you reduce the risk of any single model's bias dominating the result.
  • Loss Function Behavior: The paper highlights a critical trade-off in loss functions. While IoU and Dice losses are popular for segmentation, they tend to produce "binary" confidence maps—they are often overconfident even when they are wrong. Cross-entropy provides more nuanced, reliable confidence estimates.
Fig. 1: Left to right: FLAIR; manual annotation of a BRATS’17 subject, where yellow depicts oedema surrounding tumour core; confidence of a CNN predicting oedema, trained with cross-entropy or IoU loss. Although overall performance is similar, training with IoU (or Dice, not shown) loss alters the CNN’s behaviour, which tends to output only highly confident predictions, even when false.
  • Robustness to Preprocessing: Intensity normalization is a major source of variance in MRI analysis. EMMA handles this by including models trained on three different normalization pipelines, effectively "averaging away" the artifacts introduced by any single method.

Key Results

EMMA achieved the top position in the BRATS 2017 competition among over 50 participating teams.

  • Generalization: A standout result was EMMA's stability. While many competing methods saw a significant performance drop when moving from the validation set to the hidden test set, EMMA maintained nearly identical performance (e.g., Whole Tumor Dice of 90.1 on validation vs. 88.6 on test).
  • Quantitative Performance: On the test set, EMMA achieved Dice scores of 72.9% for the Enhancing Tumor, 88.6% for the Whole Tumor, and 78.5% for the Tumor Core.

Limitations and Future Work

  • Computational Overhead: The primary drawback is the cost of inference. Running multiple 3D CNNs (DeepMedic, FCN, and U-Net variants) is computationally expensive and time-consuming compared to a single model.
  • Consistent Failures: While ensembling corrects "inconsistent" mistakes (where only one model fails), it cannot fix "consistent" mistakes where the entire ensemble is misled by ambiguous image features.
  • Future Directions: The authors suggest using EMMA's uncertainty as an objective measure to identify which types of tumors or patients are most challenging for current AI models to learn. They also aim to explore the re-usability of EMMA across different medical imaging tasks without extensive re-tuning.

Summary updated

Abstract

Deep learning approaches such as convolutional neural nets have consistently outperformed previous methods on challenging tasks such as dense, semantic segmentation. However, the various proposed networks perform differently, with behaviour largely influenced by architectural choices and training settings. This paper explores Ensembles of Multiple Models and Architectures (EMMA) for robust performance through aggregation of predictions from a wide range of methods. The approach reduces the influence of the meta-parameters of individual models and the risk of overfitting the configuration to a particular database. EMMA can be seen as an unbiased, generic deep learning model which is shown to yield excellent performance, winning the first position in the BRATS 2017 competition among 50+ participating teams.
BibTeX
@inproceedings{kamnitsas2017ensembles,
  author = {Konstantinos Kamnitsas and Wenjia Bai and Enzo Ferrante and Steven G. McDonagh and Matthew Sinclair and Nick Pawlowski and Martin Rajchl and Matthew C. H. Lee and Bernhard Kainz and Daniel Rueckert and Ben Glocker},
  title = {{Ensembles of Multiple Models and Architectures for Robust Brain Tumour Segmentation}},
  booktitle = {BrainLes @ MICCAI},
  year = {2017},
  doi = {10.1007/978-3-319-75238-9\_38},
}
Figure