Model Ensemble for Brain Tumor Segmentation in Magnetic Resonance Imaging

Daniel Capellán-Martín, Zhifan Jiang, Abhijeet Parida, Xinyang Liu, Van Lam, Hareem Nisar, Austin Tapp, Sarah Elsharkawi, María J. Ledesma‐Carbayo, Syed Muhammad Anwar, Marius George Linguraru

TL;DR

Researchers developed a high-performing ensemble deep learning pipeline that combines the strengths of convolutional neural networks (nnU-Net) and vision transformers (Swin UNETR) to segment rare and complex brain tumors. Their approach, which included specialized post-processing to handle noise and label ambiguity, secured first place in the Pediatric Brain Tumor (PED) task of the 2023 BraTS challenge.

What Problem Does This Paper Address?

While automatic brain tumor segmentation has seen massive progress, most research has focused on adult gliomas. The 2023 Brain Tumor Segmentation (BraTS) challenge expanded this scope to include more diverse and rare cases: pediatric tumors (PED), intracranial meningiomas (MEN), and brain metastases (MET).

These categories present unique hurdles:

  1. Data Scarcity: Pediatric and metastatic cases are rarer than adult gliomas, leading to smaller training sets.
  2. Morphological Variety: Unlike adult gliomas, pediatric tumors (like DMGs) often lack clear necrosis or enhancement on MRI, making standard segmentation rules fail.
  3. Small Lesions: Metastases are often tiny; missing even one can drastically change a patient's prognosis.
  4. Label Ambiguity: In pediatric cases, many scans have "empty" ground truth labels for certain sub-regions, which can confuse standard models.

Methods

The authors proposed a two-stage pipeline: a Model Ensemble followed by Targeted Post-processing.

1. The Ensemble

Instead of relying on a single architecture, they combined two state-of-the-art models to capture different image features:

  • nnU-Net: A self-configuring CNN that excels at capturing local textures and fine details.
  • Swin UNETR: A vision transformer-based model designed to capture long-range spatial dependencies and global context.

The ensemble was performed on a region-wise basis. For example, in the PED task, they found that nnU-Net performed better on the Enhancing Tumor (ET) region, so they weighted its contributions differently during the final prediction.

2. Post-processing Strategy

The team implemented a "clean-up" phase that proved critical for their competition ranking:

  • Connected Component Analysis: They removed small, disconnected "blobs" of predicted tumor that fell below a specific voxel threshold (e.g., <130 voxels for PED), effectively reducing false positives.
  • Label Redefinition (PED-specific): Because pediatric tumors often have missing sub-regions, the authors used a threshold search based on the ratio of Enhancing Tumor (ET) to Whole Tumor (WT). If the predicted ET was too small relative to the whole mass, it was reclassified to prevent "noisy" over-segmentation.

Datasets

The study utilized the BraTS 2023 multi-parametric MRI (mpMRI) datasets, which include T1, T1-contrast enhanced (T1CE), T2, and T2-FLAIR sequences.

  • PED: 99 training / 45 validation cases.
  • MEN: 1,000 training / 141 validation cases.
  • MET: 165 training / 31 validation cases.

Key Insights and Ideas

  • CNNs + Transformers = Better Together: The "aha moment" here is that CNNs (nnU-Net) and Transformers (Swin UNETR) have complementary failure modes. By ensembling them, the researchers leveraged the CNN's precision and the Transformer's spatial awareness.
  • Post-processing is not "Cheating": In medical imaging, the raw output of a neural network is rarely the best clinical result. The authors showed that a cross-validated threshold search for removing small disconnected components significantly boosts "lesion-wise" Dice scores—a metric that penalizes small false positives heavily.
  • Task-Specific Optimization: The team didn't use a "one size fits all" approach. They discovered that Swin UNETR actually performed inferiorly on the MET task, leading them to use only nnU-Net for that specific challenge.

Key Results

The method demonstrated strong generalization across unseen test cases, resulting in the following Lesion-wise Dice Scores (a measure of overlap between prediction and ground truth, where 1.0 is perfect):

Task Enhancing Tumor (ET) Tumor Core (TC) Whole Tumor (WT) Competition Rank
PED 0.653 0.809 0.826 1st
MEN 0.876 0.867 0.849 3rd
MET 0.555 0.600 0.580 4th

The high performance in the PED task is particularly notable given the difficulty of segmenting pediatric midline gliomas.

Limitations and Future Work

  • Data Constraints: The models were trained strictly on the provided challenge data. The authors suggest that pre-training on larger, more diverse datasets could further improve robustness.
  • Synthetic Data: Given the rarity of pediatric tumors, the authors propose that Generative AI (like GANs or Diffusion Models) could be used to create synthetic training examples to help the model learn rare tumor morphologies.
  • Foundation Models: The researchers point toward self-supervised learning and "foundation models" as the next step to create a single system that generalizes across all brain tumor types without needing task-specific tuning.

Summary updated

Abstract

Segmenting brain tumors in multi-parametric magnetic resonance imaging enables performing quantitative analysis in support of clinical trials and personalized patient care. This analysis provides the potential to impact clinical decision-making processes, including diagnosis and prognosis. In 2023, the well-established Brain Tumor Segmentation (BraTS) challenge presented a substantial expansion with eight tasks and 4,500 brain tumor cases. In this paper, we present a deep learning-based ensemble strategy that is evaluated for newly included tumor cases in three tasks: pediatric brain tumors (PED), intracranial meningioma (MEN), and brain metastases (MET). In particular, we ensemble outputs from state-of-the-art nnU-Net and Swin UNETR models on a region-wise basis. Furthermore, we implemented a targeted post-processing strategy based on a cross-validated threshold search to improve the segmentation results for tumor sub-regions. The evaluation of our proposed method on unseen test cases for the three tasks resulted in lesion-wise Dice scores for PED: 0.653, 0.809, 0.826; MEN: 0.876, 0.867, 0.849; and MET: 0.555, 0.6, 0.58; for the enhancing tumor, tumor core, and whole tumor, respectively. Our method was ranked first for PED, third for MEN, and fourth for MET, respectively.
BibTeX
@misc{capellanmartin2024model,
  author = {Daniel Capellán-Martín and Zhifan Jiang and Abhijeet Parida and Xinyang Liu and Van Lam and Hareem Nisar and Austin Tapp and Sarah Elsharkawi and María J. Ledesma‐Carbayo and Syed Muhammad Anwar and Marius George Linguraru},
  title = {{Model Ensemble for Brain Tumor Segmentation in Magnetic Resonance Imaging}},
  year = {2024},
  eprint = {2409.08232},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.08232},
}
Figure