Abstract
Accurate medical image segmentation commonly requires effective learning of the complementary information from multimodal data. However, in clinical practice, we often encounter the problem of missing imaging modalities. We tackle this challenge and propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code, which uniquely sticks to each modality, and the modality-invariant content code, which absorbs multimodal information for the segmentation task. With enhanced modality-invariance, the disentangled content code from each modality is fused into a shared representation which gains robustness to missing data. The fusion is achieved via a learning-based strategy to gate the contribution of different modalities at different locations. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset. With competitive performance to the state-of-the-art approaches for full modality, our method achieves outstanding robustness under various missing modality(ies) situations, significantly exceeding the state-of-the-art method by over 16% in average for Dice on whole tumor segmentation.
Introduction
Accurate brain tumor segmentation typically relies on four distinct MRI modalities: FLAIR, T1, T1 contrast-enhanced (T1c), and T2. Each provides unique biological information; for instance, T1c highlights the enhancing tumor, while FLAIR is superior for identifying edema. In clinical reality, however, a full set of these scans is often unavailable due to time constraints, patient conditions, or varying scanning protocols.
Most existing segmentation models are "brittle," degrading significantly if an expected input is missing. Previous attempts to solve this, such as the HeMIS (Hetero-modal Image Segmentation) framework, used simple averaging or max-pooling to fuse features, which often fails to capture the nuanced importance of different modalities at different spatial locations. This paper introduces a framework designed for robustness through explicit feature disentanglement and learned spatial gating.
Methodology
The proposed architecture employs a two-pronged strategy to achieve robustness:
- Feature Disentanglement: The network decomposes each input modality into two separate latent spaces. The modality-specific appearance code captures the "style" or intensity distribution unique to that specific MRI sequence, while the modality-invariant content code captures the underlying anatomical structure (the tumor) that should be consistent across all sequences. To ensure effective disentanglement, the model utilizes a pseudo-cycle-consistency loss, attempting to reconstruct the original input image by combining the fused content code with the specific appearance code.
- Gated Feature Fusion: Rather than simple averaging, the model learns gating weight maps. These maps allow the network to decide, voxel-by-voxel, which modality should contribute most to the final representation. For example, the gate might prioritize T1c features in the region of the enhancing tumor core while prioritizing FLAIR for the surrounding edema.
- Modality Dropout: During training, the content codes of certain modalities are randomly "dropped" (set to zero). This forces the network to learn how to produce high-quality segmentation and reconstruction even when the input is incomplete.
Results and Discussion
The model was evaluated using the BRATS 2015 dataset, consisting of 274 training cases and 110 test cases. The task involved segmenting three nested regions: Complete tumor, Tumor core, and Enhancing tumor.
Key Findings
- Robustness to Missing Data: In scenarios where modalities are missing, the proposed method outperformed the state-of-the-art (HeMIS) by an average of 16% in Dice score for whole tumor segmentation. As shown in the qualitative results, the model maintains a coherent segmentation mask even when reduced to a single T1 scan.
- Full Modality Performance: When all four modalities are present, the model remains competitive with top-tier specialized architectures, achieving a Dice score of 84% for the complete tumor.
- Ablation Success: Experimental results proved that both disentanglement and gated fusion are necessary; removing either component led to a significant drop in robustness across the 15 possible combinations of available modalities.
Conclusion
By explicitly stripping away modality-specific "appearance" and employing spatial awareness in fusion, this framework achieves graceful degradation in the face of missing data. While the framework is computationally intensive due to independent encoders and multiple decoders, it sets a new benchmark for robust multimodal medical image analysis. Future work may extend this disentanglement approach to other tasks such as cross-modality synthesis or deformable image registration.
Summary updated