Anatomy‐Guided, Modality‐Agnostic Segmentation of Neuroimaging Abnormalities

Diala Lteif, Divya Appapogu, Sarah Adel Bargal, Bryan A. Plummer, Vijaya B. Kolachalama

TL;DR

This paper introduces Region ModalMix (RMM), a novel data augmentation strategy that makes brain MRI segmentation models significantly more robust to missing imaging sequences. By using anatomical "maps" to mix different MRI modalities within specific brain structures during training, the framework enables AI to maintain high diagnostic accuracy even when critical data (like a T1 or FLAIR scan) is unavailable.

What Problem Does This Paper Address?

In a perfect world, every patient undergoing a brain MRI would receive a full suite of sequences (T1, T2, T1ce, and FLAIR), providing a comprehensive view of anatomy and pathology. In reality, clinical constraints—such as patient discomfort, emergency time limits, or high costs—often result in "modality-incomplete" datasets where one or more sequences are missing.

Most deep learning models for medical imaging are "brittle"; they are trained on complete data and their performance collapses when faced with missing inputs. While some researchers try to "hallucinate" (synthesize) missing scans or use complex fusion layers, these methods often fail to capture the full clinical variability or require massive amounts of data to train. This paper seeks a way to make models modality-agnostic—capable of identifying abnormalities regardless of which specific subset of MRI sequences is provided.

Methods

The core innovation is Region ModalMix (RMM). Unlike standard data augmentation techniques like "CutMix," which stitches together random cuboid patches from different images, RMM is anatomy-guided.

  1. Anatomical Parcellation: The researchers first divide the brain into 101 distinct anatomical regions (using a tool called SynthSeg+).
  2. Region-Based Mixing: During training, the algorithm randomly selects these anatomical regions and swaps "information" across different MRI modalities for the same subject. For example, it might take the visual features of the thalamus from a T2 scan and "mix" them into a FLAIR scan's representation.
  3. Structural Coherence: Because the mixing happens within predefined anatomical boundaries, the resulting "augmented" images remain biologically plausible. This forces the model to learn features that are consistent across modalities rather than relying on the specific "look" of a single sequence.
  4. Model-Agnostic Framework: RMM is designed to be a "plug-and-play" strategy. The authors tested it across several state-of-the-art architectures, including transformer-based models (like mmFormer and M2FTrans) and convolutional networks (like RFNet).

Datasets

The study utilized the Multimodal Brain Tumor Segmentation Challenge (BraTS) 2020 dataset.

  • Size: 369 subjects (293 with high-grade gliomas and 76 with low-grade gliomas).
  • Modalities: Every subject had four standard MRI sequences: T1, T1-contrast enhanced (T1ce), T2, and FLAIR.
  • Task: Segmenting three tumor subregions: the necrotic core, edema, and the enhancing tumor.

To test the "missing modality" problem, the researchers simulated scenarios where 1, 2, or 3 sequences were randomly removed during evaluation.

Key Insights and Ideas

  • Anatomy as a Prior: The "aha moment" here is that the brain’s structure is a constant. By forcing the model to see different "textures" (modalities) within the same "container" (anatomical region), the model learns that a tumor is a tumor, whether it's glowing on a FLAIR scan or dark on a T1.
  • Beyond Random Patches: The authors show that random cuboid mixing (3D MM-CutMix) creates "anatomically incoherent boundaries" that can confuse a model. RMM’s boundary-aware mixing provides a much cleaner signal for the neural network to learn from.
  • The "Full Data" Bonus: Surprisingly, RMM doesn't just help when data is missing; it also improves performance when all modalities are present. This suggests that the anatomical mixing acts as a powerful regularizer, helping the model generalize better overall.

Key Results

The RMM framework consistently outperformed state-of-the-art methods across 15 different modality combinations.

  • Boundary Precision: In the most difficult cases (only one modality available), RMM reduced the 95th percentile Hausdorff Distance (HD95)—a measure of boundary error—by an average of 9.68 mm.
  • Overlap Accuracy: It achieved a 1.36% improvement in the Dice Similarity Coefficient (a measure of spatial overlap) over baseline models in single-modality settings.
  • Backbone Synergy: The best results were seen when RMM was paired with the M2FTrans backbone, particularly in segmenting the "Enhancing Tumor" (ET) and "Tumor Core" (TC), which are often the most difficult regions to delineate.

Limitations and Future Work

While promising, the authors acknowledge a few hurdles:

  • Dependency on Pre-processing: RMM currently relies on pre-computed anatomical parcellations. If the initial "map" of the brain is inaccurate (e.g., in pediatric cases or highly distorted anatomies), the augmentation might be less effective.
  • Static Augmentation: The current version uses a fixed set of rules for mixing. Future iterations could involve "learnable" augmentations that adapt to specific pathologies.
  • Modality Scope: The study focused on standard structural MRI. Future work is needed to see if this approach holds up for more specialized sequences like diffusion-weighted imaging (DWI) or susceptibility-weighted imaging (SWI).
  • Real-world Validation: While the "missingness" was simulated here, the next step is testing on large-scale clinical datasets where data is naturally missing due to real-world hospital workflows.

Summary updated

Abstract

Magnetic resonance imaging (MRI) offers multiple sequences that provide complementary views of brain anatomy and pathology. However, real-world datasets often exhibit variability in sequence availability due to clinical and logistical constraints. This variability complicates radiological interpretation and limits the generalizability of machine learning models that depend on a consistent multimodal input. Here, we propose an anatomy-guided, modality-agnostic framework to assess disease-related abnormalities in brain MRI, leveraging structural context to ensure robustness in diverse input configurations. Central to our approach is Region ModalMix (RMM), an augmentation strategy that integrates anatomical priors during training to improve model performance under missing or variable modality conditions. Using the BraTS 2020 dataset (n = 369), our framework outperformed state-of-the-art methods, achieving a 9.68 mm average reduction in 95th percentile Hausdorff Distance (HD95) and a 1.36 percentage point improvement in Dice Similarity Coefficient (DSC) over baselines with only one available modality. To evaluate out-of-distribution generalization, we tested RMM on the MU-Glioma-Post dataset (n = 593), which includes heterogeneous post-operative glioma cases. Despite distribution shifts, RMM maintained strong performance, reducing HD95 by 18.24 mm and improving DSC by 9.54% points in the most severe missing-modality scenario. Our framework is applicable to multimodal neuroimaging pipelines, enabling more generalizable abnormality detection under heterogeneous data availability.
BibTeX
@inproceedings{lteif2025anatomyguided,
  author = {Diala Lteif and Divya Appapogu and Sarah Adel Bargal and Bryan A. Plummer and Vijaya B. Kolachalama},
  title = {{Anatomy‐Guided, Modality‐Agnostic Segmentation of Neuroimaging Abnormalities}},
  booktitle = {Human},
  year = {2025},
  doi = {10.1002/hbm.70329},
}

Loading related content...

Figure