TL;DR
The EMMA (Ensembles of Multiple Models and Architectures) framework tackles the inherent bias and sensitivity of deep learning models by aggregating predictions from a diverse set of 3D CNN architectures. By "averaging away" the influence of specific meta-parameters and training configurations, the authors created a highly robust system that won first place in the prestigious BRATS 2017 brain tumor segmentation challenge.
What Problem Does This Paper Address?
Deep learning has revolutionized medical image segmentation, but individual models are notoriously finicky. A network's performance is heavily influenced by "meta-parameters"—architectural choices (like receptive field size), loss functions (Dice vs. Cross-Entropy), and preprocessing steps (intensity normalization).
This sensitivity creates two major problems:
- Overfitting to Configurations: A model manually tuned to perform perfectly on one dataset might fail on another because its specific configuration is over-fit to the training distribution.
- Biased Analysis: If a model is sensitive to specific textures due to its architecture, any clinical findings derived from its segmentations are inherently biased by those architectural choices.
The authors argue that we need "objective" models that are invariant to these arbitrary configuration choices.
Methods
The core contribution is EMMA, which treats meta-parameters as stochastic variables and marginalizes their effect through ensembling. Unlike typical ensembles that use the same model with different random seeds, EMMA combines fundamentally different architectures:
- DeepMedic: A multi-scale 3D CNN that processes context at different resolutions in parallel.
- FCN (Fully Convolutional Networks): 3D versions including residual-based variants.
- U-Net: 3D adaptations using both summation and concatenation for skip connections.
Beyond architectural diversity, the models were trained using different loss functions (Cross-Entropy, IoU, and Dice) and different intensity normalization schemes. At test time, the final segmentation is produced by simply averaging the class-confidence maps from all individual models.
Datasets
The system was evaluated using the BRATS 2017 (Brain Tumour Segmentation Challenge) dataset:
- Training Set: 285 cases (210 High Grade Glioma, 75 Low Grade Glioma) with manual annotations for necrotic core, edema, and enhancing tumor.
- Validation Set: 46 cases.
- Test Set: 146 cases (used for the final competition ranking).
- Modalities: Each case includes four MRI sequences: FLAIR, T1, T1-contrast enhanced (T1ce), and T2.
Key Insights and Ideas
- Ensembling for Objectiveness: The authors frame ensembling not just as a performance booster, but as a way to achieve "configuration-invariance." By covering a subspace of high-quality models, you reduce the risk of any single model's bias dominating the result.
- Loss Function Behavior: The paper highlights a critical trade-off in loss functions. While IoU and Dice losses are popular for segmentation, they tend to produce "binary" confidence maps—they are often overconfident even when they are wrong. Cross-entropy provides more nuanced, reliable confidence estimates.
- Robustness to Preprocessing: Intensity normalization is a major source of variance in MRI analysis. EMMA handles this by including models trained on three different normalization pipelines, effectively "averaging away" the artifacts introduced by any single method.
Key Results
EMMA achieved the top position in the BRATS 2017 competition among over 50 participating teams.
- Generalization: A standout result was EMMA's stability. While many competing methods saw a significant performance drop when moving from the validation set to the hidden test set, EMMA maintained nearly identical performance (e.g., Whole Tumor Dice of 90.1 on validation vs. 88.6 on test).
- Quantitative Performance: On the test set, EMMA achieved Dice scores of 72.9% for the Enhancing Tumor, 88.6% for the Whole Tumor, and 78.5% for the Tumor Core.
Limitations and Future Work
- Computational Overhead: The primary drawback is the cost of inference. Running multiple 3D CNNs (DeepMedic, FCN, and U-Net variants) is computationally expensive and time-consuming compared to a single model.
- Consistent Failures: While ensembling corrects "inconsistent" mistakes (where only one model fails), it cannot fix "consistent" mistakes where the entire ensemble is misled by ambiguous image features.
- Future Directions: The authors suggest using EMMA's uncertainty as an objective measure to identify which types of tumors or patients are most challenging for current AI models to learn. They also aim to explore the re-usability of EMMA across different medical imaging tasks without extensive re-tuning.
Summary updated