Disentangle, Align and Fuse for Multimodal and Semi-Supervised Image Segmentation

Agisilaos Chartsias, Giorgos Papanastasiou, Chengjia Wang, Scott Semple, David E. Newby, Rohan Dharmakumar, Sotirios A. Tsaftaris

TL;DR

The authors introduce DAFNet, a novel framework that improves medical image segmentation by disentangling images into shared anatomical and modality-specific factors. By non-linearly aligning and fusing these anatomical representations, the model achieves high accuracy even when one imaging modality has few or zero manual annotations.

What Problem Does This Paper Address?

In clinical practice, doctors often use multiple MRI sequences (modalities) to get a full picture of a patient's health. For example, "cine-MR" shows heart motion clearly, while "LGE-MR" is better for identifying scarred tissue.

However, automated segmentation tools usually treat these modalities in isolation. Combining them is notoriously difficult because:

  1. Misalignment: Organs move (especially the heart), and different scans have different spatial resolutions.
  2. Intensity Differences: The same tissue looks completely different across sequences.
  3. Label Scarcity: Expert annotations are expensive. Often, we have plenty of labels for one common modality (like cine-MR) but very few for another (like LGE).

The authors aim to bridge this gap by creating a system that can "borrow" anatomical knowledge from one modality to help segment another, even without direct supervision.

Methods

The core of the proposed Disentangle, Align and Fuse Network (DAFNet) is a three-step process:

  1. Disentanglement: The network uses separate encoders to decompose an input image into two parts: an anatomical factor (spatial maps representing structure) and a modality factor (a vector capturing style/intensity). To ensure these are truly separated, the anatomy factor is represented as discrete categorical maps.
  2. Alignment: Since images from different scans are rarely perfectly aligned, DAFNet uses a Spatial Transformer Network (STN). This module learns to non-linearly warp the anatomical factors from one modality to match the other.
  3. Fusion: The aligned anatomical factors are combined using a pixel-wise "max-fusion" operator. This creates a single, high-quality "fused anatomy" that contains the best information from both inputs.
The DAFNet schematic shows how LGE and cine-MR inputs are disentangled, aligned, and fused to produce a final segmentation.

To train with limited labels, the model uses a reconstruction objective. It tries to "re-entangle" the factors to recreate the original images, serving as a form of self-supervision.

Datasets

The authors evaluated DAFNet on three distinct multimodal datasets:

  • ERI Dataset: Cardiac cine-MR and LGE-MR images from 28 patients.
  • CHAOS Dataset: Abdominal T1-dual inphase and T2-SPIR MRI scans for liver, kidney, and spleen segmentation.
  • BOLD Dataset: Cardiac cine-MR and CP-BOLD images, focusing on myocardial ischemia.

Key Insights and Ideas

  • Anatomy as a Universal Language: By forcing the network to output categorical "anatomy maps," the authors create a modality-invariant representation. This allows a segmentor trained on one modality to work on another simply by projecting both into this shared space.
  • Papillary Muscles Matter: A key "aha" moment was realizing that anatomical factors shouldn't just be segmentations. As shown in Figure 4, the anatomy factors capture fine details (like papillary muscles) that are necessary for image reconstruction but might be ignored in a standard segmentation mask.
  • Dynamic Pairing: In real-world data, slices might not match up perfectly in time or space. The authors introduced a "non-expert pairing" loss that automatically weights the most informative image pairs during training, reducing the need for tedious manual data prep.
Cine and LGE anatomy factors. Note how the same anatomical regions appear in the same channels across modalities.

Key Results

DAFNet consistently outperformed single-modality models and existing multimodal benchmarks (like SDNet or DualStream):

  • Zero-Annotation Performance: Most impressively, when the model was given zero labels for the LGE modality, it still achieved a Dice score of ~74% by leveraging cine-MR labels. Baselines often failed completely or performed significantly worse in this "zero-shot" scenario.
  • Full Supervision: With 100% labels, DAFNet reached a Dice score of 82% on LGE data, showing that multimodal fusion provides a significant boost over single-modality U-Nets (~78%).
  • Robustness: The inclusion of the STN alignment module was critical; without it, performance dropped by 7-10% in low-data regimes.

Limitations and Future Work

  • 2D vs. 3D: The current implementation is 2D. While this is computationally efficient, extending the framework to 3D could better capture volumetric consistency.
  • Pathology Modeling: The model assumes anatomy is shared. In cases of extreme pathology (e.g., a large tumor visible in only one modality), the "shared anatomy" assumption might need to be relaxed or refined.
  • Future Directions: The authors suggest exploring more complex factorizations that could specifically isolate pathological features or imaging artifacts from the underlying healthy anatomy.

Reference: Chartsias, A., et al. "Disentangle, align and fuse for multimodal and semi-supervised image segmentation." arXiv:1911.04417v5, 2020.

Summary updated

Abstract

Magnetic resonance (MR) protocols rely on several sequences to assess pathology and organ status properly. Despite advances in image analysis, we tend to treat each sequence, here termed modality, in isolation. Taking advantage of the common information shared between modalities (an organ's anatomy) is beneficial for multi-modality processing and learning. However, we must overcome inherent anatomical misregistrations and disparities in signal intensity across the modalities to obtain this benefit. We present a method that offers improved segmentation accuracy of the modality of interest (over a single input model), by learning to leverage information present in other modalities, even if few (semi-supervised) or no (unsupervised) annotations are available for this specific modality. Core to our method is learning a disentangled decomposition into anatomical and imaging factors. Shared anatomical factors from the different inputs are jointly processed and fused to extract more accurate segmentation masks. Image misregistrations are corrected with a Spatial Transformer Network, which non-linearly aligns the anatomical factors. The imaging factor captures signal intensity characteristics across different modality data and is used for image reconstruction, enabling semi-supervised learning. Temporal and slice pairing between inputs are learned dynamically. We demonstrate applications in Late Gadolinium Enhanced (LGE) and Blood Oxygenation Level Dependent (BOLD) cardiac segmentation, as well as in T2 abdominal segmentation. Code is available at https://github.com/vios-s/multimodal_segmentation.
BibTeX
@article{chartsias2021disentangle,
  author = {Agisilaos Chartsias and Giorgos Papanastasiou and Chengjia Wang and Scott Semple and David E. Newby and Rohan Dharmakumar and Sotirios A. Tsaftaris},
  title = {{Disentangle, Align and Fuse for Multimodal and Semi-Supervised Image Segmentation}},
  journal = {IEEE Trans. Medical Imaging},
  year = {2021},
  url = {https://openreview.net/forum?id=cvfkSe9RbZ},
}

Loading related content...

Figure