Abstract
Multimodal medical imaging, such as combining cine-MR and LGE-MR, provides comprehensive clinical insights but presents significant challenges for automated segmentation due to spatial misalignment, intensity variations, and the scarcity of expert annotations. We introduce DAFNet (Disentangle, Align and Fuse Network), a novel framework that decomposes images into modality-invariant anatomical factors and modality-specific style factors. By non-linearly aligning these anatomical representations and fusing them, DAFNet achieves state-of-the-art performance in multimodal segmentation. Notably, the framework demonstrates strong "zero-shot" capabilities, accurately segmenting modalities for which no manual labels were provided during training.
Introduction
In cardiology and other medical fields, multiple MRI sequences are often acquired to visualize different tissue properties. For instance, cine-MR captures cardiac motion, while Late Gadolinium Enhancement (LGE) MRI is the gold standard for identifying myocardial scarring. However, integrating these modalities into a unified segmentation pipeline is difficult. Organs move between scans, spatial resolutions vary, and tissue intensities differ drastically. Furthermore, while labels might be abundant for common sequences like cine-MR, they are often scarce for specialized sequences like LGE.
DAFNet addresses these issues by creating a shared anatomical space where information from different modalities can be aligned and fused, allowing the model to leverage knowledge from well-labeled datasets to assist in segmenting sparsely labeled or unlabeled modalities.
Methodology: The DAFNet Framework
The core innovation of DAFNet is a three-stage process: Disentanglement, Alignment, and Fusion.
1. Disentanglement
The network employs separate encoders to decompose an input image into two distinct components:
- Anatomical Factor ($s$): A set of spatial maps representing the underlying structure. These are represented as discrete categorical maps to ensure they remain modality-invariant.
- Modality Factor ($z$): A low-dimensional vector capturing the specific "style" or intensity characteristics of the imaging sequence.
2. Alignment and Fusion
Because scans are rarely perfectly registered, DAFNet utilizes a Spatial Transformer Network (STN) to learn a non-linear warping that aligns the anatomical factors of one modality to another. Once aligned, a pixel-wise "max-fusion" operator combines these factors into a single, high-quality "fused anatomy" representation that inherits the best features of both inputs.
Training and Cross-Modality Synthesis
To train effectively with limited supervision, DAFNet uses a reconstruction-based self-supervision strategy. The model attempts to "re-entangle" the factors—combining the anatomy of one image with the modality factor of another—to synthesize realistic cross-modality images. This architecture allows for zero-shot segmentation: a segmentor trained on the anatomical factors of cine-MR can be applied directly to the anatomical factors of LGE-MR, as both are projected into the same modality-invariant space.
Experimental Results
The authors evaluated DAFNet on three multimodal datasets: cardiac (ERI and BOLD) and abdominal (CHAOS).
Key Findings
- Zero-Shot Success: When provided with zero labels for the LGE modality, DAFNet achieved a Dice score of approximately 74% by leveraging cine-MR labels. This significantly outperforms traditional methods that typically fail in the absence of target-domain labels.
- Superior Full Supervision: With 100% labels, DAFNet reached a Dice score of 82% on LGE data, surpassing single-modality U-Nets (78%) and other multimodal benchmarks.
- Anatomical Fidelity: A key insight is that learned anatomical factors capture fine structural details (such as papillary muscles) necessary for accurate image reconstruction, even if those details are not explicitly present in the final segmentation mask.
- Alignment Robustness: The STN module proved essential; removing it led to a 7-10% performance drop in low-data scenarios, highlighting the importance of handling spatial misalignment.
Conclusion
DAFNet provides a robust framework for multimodal medical image segmentation by treating anatomy as a universal, modality-invariant language. By disentangling structure from style and explicitly modeling spatial alignment, it enables effective information sharing across imaging sequences. Future work will explore extending this 2D framework to 3D volumes and refining the model to handle extreme pathologies where anatomical features might not be shared across all modalities.
Reference: Chartsias, A., et al. "Disentangle, align and fuse for multimodal and semi-supervised image segmentation." arXiv:1911.04417v5, 2020.
Summary updated