Introduction
Brain tumor segmentation using multi-modal MRI—specifically T1, T2, T1ce, and Flair sequences—is essential for precise diagnosis and surgical planning. In clinical practice, however, datasets are rarely complete. Scans are frequently missing due to patient motion, imaging artifacts, or time constraints. While existing AI models for "incomplete modality" segmentation often assume a balanced missing rate (where every scan type is equally likely to be absent), real-world clinical data is typically imbalanced. For instance, T1 scans are almost always available, while T2 sequences are more prone to issues and are frequently missing.
Current segmentation models struggle with this imbalance, often failing to maintain "modality invariance"—the ability to produce consistent results regardless of which specific scans are present. To address this, we introduce Koopman Multi-modality Decomposition (KMD), a novel plug-and-play module that leverages Koopman theory to disentangle shared and modality-specific information, significantly improving performance in realistic, imbalanced clinical scenarios.
Methodology: Koopman Multi-modality Decomposition
The KMD module is built upon Koopman Invariant Subspace theory, which suggests that complex non-linear dynamics—such as the relationships between different MRI modalities—can be represented as linear mappings within a high-dimensional latent space.
As illustrated in the architecture above, KMD processes features through several critical stages:
- Feature Disentanglement: The module splits latent features into modality-common components, which represent shared anatomical structures, and modality-specific components, which capture the unique characteristics of a particular scan type.
- Koopman Invariant Subspace: By identifying a subspace where modalities relate linearly ($Ax + b$), the model can effectively "infer" or reconstruct relationships even when certain data is missing. This bi-directional mapping allows information to flow between any two modalities, reducing bias toward the most abundant scan types.
- Stability via Spherization: Before decomposition, the module applies centralization and spherization to normalize features. This ensures the model focuses on the underlying relationships between features rather than their absolute scales or biases.
The module is optimized using a three-part loss function:
- Koopman Loss ($L_k$): Ensures that the linear mappings between modality pairs remain consistent.
- Common Loss ($L_c$): Uses cosine similarity to align shared information across all available modalities.
- Orthogonality Loss ($L_o$): Ensures that modality-specific information does not "leak" into the shared feature set.
Experimental Results and Insights
The researchers evaluated KMD on two gold-standard benchmarks: BraTS2018 (for balanced missing rates) and BraTS2020 (for imbalanced scenarios). KMD was integrated into several state-of-the-art backbones, including RFNet, mmFormer, and M2FTrans, proving its versatility as a plug-and-play component.
Performance and Robustness
KMD consistently outperformed existing methods across various scenarios. On the BraTS2020 dataset, KMD showed significant gains over the previous state-of-the-art, particularly in the "Enhancing Tumor" (ET) class—notoriously the most difficult region to segment. By linearizing the complex relationships between modalities, KMD allows the neural network to focus more effectively on anatomical consistency.
Latent Space Visualization
Qualitative analysis using t-SNE plots confirms the effectiveness of the KMD module in organizing the latent space.
As shown in the visualizations, KMD produces much clearer boundaries between tumor regions in the latent space compared to baseline models. By successfully separating shared features from modality-specific ones, the model achieves a higher degree of modality invariance, leading to more robust segmentation results regardless of which MRI sequences are missing.
Conclusion
KMD provides a robust, "plug-and-play" solution for brain tumor segmentation in real-world clinical environments where MRI modalities are often missing in an imbalanced fashion. While the module requires careful balancing of its three loss components, its ability to improve the performance of various existing architectures makes it a highly practical tool. Future work may extend this Koopman-based decomposition to other multi-modal tasks, such as combining imaging data with genomic or clinical text information.
Summary updated