Abstract
The use of multimodal data in assisted diagnosis and segmentation has emerged as a prominent area of interest in current research. However, one of the primary challenges is how to effectively fuse multimodal features. Most current approaches focus on the integration of multimodal features while ignoring the correlation and consistency between different modal features, leading to the inclusion of potentially irrelevant information. To address this issue, we introduce an innovative Multimodal Information Cross Transformer (MicFormer), which employs a dual-stream architecture to simultaneously extract features from each modality. Leveraging the Cross Transformer, it queries features from one modality and retrieves corresponding responses from another, facilitating effective communication between bimodal features. Additionally, we incorporate a deformable Transformer architecture to expand the search space. We conducted experiments on the MM-WHS dataset, and in the CT-MRI multimodal image segmentation task, we successfully improved the whole-heart segmentation DICE score to 85.57 and MIoU to 75.51. Compared to other multimodal segmentation techniques, our method outperforms by margins of 2.83 and 4.23, respectively.
Introduction
In medical AI, combining complementary imaging modalities like CT (excellent for bone and dense structures) and MRI (superior for soft tissue) is essential for high-precision diagnosis and surgical planning. Creating a "super-view" from these disparate sources, however, presents significant technical hurdles. Effectively fusing these modalities requires more than just stacking data; it requires an intelligent interaction mechanism that preserves the unique strengths of each while filtering out irrelevant noise.
The Challenges of Multimodal Fusion
Current research in multimodal medical image segmentation often falls into two primary "traps":
- The "Unified Input" Trap: Many models simply concatenate CT and MRI images as a single input. This forces the network to manage fusion internally without explicit guidance, often resulting in "feature distortion" where noise from one modality obscures useful signals in the other.
- The "Fixed Window" Trap: Standard architectures like Swin Transformers rely on fixed spatial windows for attention. Because patients may shift between scans or because different modalities represent anatomy with varying spatial characteristics, features do not always align perfectly. A rigid window often fails to capture corresponding features across modalities.
The MicFormer Architecture
To overcome these limitations, we propose MicFormer (Multimodal Information Cross Transformer). The network is built upon a dual-stream, U-shaped parallel architecture. Instead of merging inputs early, MicFormer maintains two separate paths—one for CT and one for MRI—to extract high-quality unimodal features before any interaction occurs.
The architecture employs a series of Swin Transformer and Cross Transformer blocks. The dual-stream approach ensures that the unique physical characteristics of each imaging type are preserved throughout the encoding process, merging them only where semantic relevance is established.
Technical Innovations: Deformable Cross Attention
The core of MicFormer's interaction capability lies in the Cross Transformer Block and the Deformable Operator. In the Cross Transformer, one modality (e.g., MRI) acts as a "query" to retrieve relevant information from the other (e.g., CT), which serves as the "key" and "value."
The Deformable Operator is a novel addition that addresses spatial misalignment. Instead of sampling from a fixed grid, the model calculates a displacement vector (offset). This allows the attention mechanism to dynamically "reach out" and sample features from the most relevant neighboring voxels, effectively creating a deformable receptive field that adapts to the underlying anatomical data.
Experimental Results
We evaluated MicFormer on the MM-WHS (Multi-Modality Whole Heart Segmentation) dataset, consisting of 20 cardiac MRI/CT pairs. The images were preprocessed using the SyN algorithm for initial alignment and cropped to regions of interest.
MicFormer established a new state-of-the-art for this benchmark:
- DICE Score: 85.57 (an improvement of 2.83 points over previous methods).
- MIoU (Mean Intersection over Union): 75.51 (a margin of 4.23 points over competitors).
Qualitative analysis indicates that MicFormer produces significantly cleaner segmentation boundaries and fewer artifacts compared to baseline models like VT-Unet or Swin-Unet.
Conclusion and Future Work
MicFormer demonstrates that treating multimodal fusion as a reciprocal conversation—rather than simple concatenation—yields superior results. By moving away from fixed-window constraints and preserving unimodal integrity through dual streams, the model handles spatial variability and feature distortion more effectively.
Future work will focus on improving the 95% Hausdorff Distance (HD95) metric, where MicFormer currently trails slightly behind models with stronger inductive biases. Additionally, we aim to test the architecture on larger, more diverse datasets to further validate its clinical utility.
The implementation of MicFormer is available at https://github.com/fxxJuses/MICFormer.
Summary updated