KMD: Koopman Multi-modality Decomposition for Generalized Brain Tumor Segmentation under Incomplete Modalities

Tianyi Liu, Haochuan Jiang, Kaizhu HuangCVPR 2025 · DOI

Introduction

Brain tumor segmentation using multi-modal MRI—specifically T1, T2, T1ce, and Flair sequences—is essential for precise diagnosis and surgical planning. In clinical practice, however, datasets are rarely complete. Scans are frequently missing due to patient motion, imaging artifacts, or time constraints. While existing AI models for "incomplete modality" segmentation often assume a balanced missing rate (where every scan type is equally likely to be absent), real-world clinical data is typically imbalanced. For instance, T1 scans are almost always available, while T2 sequences are more prone to issues and are frequently missing.

Current segmentation models struggle with this imbalance, often failing to maintain "modality invariance"—the ability to produce consistent results regardless of which specific scans are present. To address this, we introduce Koopman Multi-modality Decomposition (KMD), a novel plug-and-play module that leverages Koopman theory to disentangle shared and modality-specific information, significantly improving performance in realistic, imbalanced clinical scenarios.

Methodology: Koopman Multi-modality Decomposition

The KMD module is built upon Koopman Invariant Subspace theory, which suggests that complex non-linear dynamics—such as the relationships between different MRI modalities—can be represented as linear mappings within a high-dimensional latent space.

Detailed architecture of the KMD module, illustrating feature disentanglement and the Koopman-based loss functions.

As illustrated in the architecture above, KMD processes features through several critical stages:

  • Feature Disentanglement: The module splits latent features into modality-common components, which represent shared anatomical structures, and modality-specific components, which capture the unique characteristics of a particular scan type.
  • Koopman Invariant Subspace: By identifying a subspace where modalities relate linearly ($Ax + b$), the model can effectively "infer" or reconstruct relationships even when certain data is missing. This bi-directional mapping allows information to flow between any two modalities, reducing bias toward the most abundant scan types.
  • Stability via Spherization: Before decomposition, the module applies centralization and spherization to normalize features. This ensures the model focuses on the underlying relationships between features rather than their absolute scales or biases.

The module is optimized using a three-part loss function:

  1. Koopman Loss ($L_k$): Ensures that the linear mappings between modality pairs remain consistent.
  2. Common Loss ($L_c$): Uses cosine similarity to align shared information across all available modalities.
  3. Orthogonality Loss ($L_o$): Ensures that modality-specific information does not "leak" into the shared feature set.

Experimental Results and Insights

The researchers evaluated KMD on two gold-standard benchmarks: BraTS2018 (for balanced missing rates) and BraTS2020 (for imbalanced scenarios). KMD was integrated into several state-of-the-art backbones, including RFNet, mmFormer, and M2FTrans, proving its versatility as a plug-and-play component.

Performance and Robustness

KMD consistently outperformed existing methods across various scenarios. On the BraTS2020 dataset, KMD showed significant gains over the previous state-of-the-art, particularly in the "Enhancing Tumor" (ET) class—notoriously the most difficult region to segment. By linearizing the complex relationships between modalities, KMD allows the neural network to focus more effectively on anatomical consistency.

Latent Space Visualization

Qualitative analysis using t-SNE plots confirms the effectiveness of the KMD module in organizing the latent space.

t-SNE visualizations demonstrating improved feature separation and modality invariance with KMD.

As shown in the visualizations, KMD produces much clearer boundaries between tumor regions in the latent space compared to baseline models. By successfully separating shared features from modality-specific ones, the model achieves a higher degree of modality invariance, leading to more robust segmentation results regardless of which MRI sequences are missing.

Conclusion

KMD provides a robust, "plug-and-play" solution for brain tumor segmentation in real-world clinical environments where MRI modalities are often missing in an imbalanced fashion. While the module requires careful balancing of its three loss components, its ability to improve the performance of various existing architectures makes it a highly practical tool. Future work may extend this Koopman-based decomposition to other multi-modal tasks, such as combining imaging data with genomic or clinical text information.

Summary updated

BibTeX
@inproceedings{liu2025kmd,
  author = {Tianyi Liu and Haochuan Jiang and Kaizhu Huang},
  title = {{KMD: Koopman Multi-modality Decomposition for Generalized Brain Tumor Segmentation under Incomplete Modalities}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.01460},
}
Figure