Influence of early through late fusion on pancreas segmentation from imperfectly registered multimodal magnetic resonance imaging

Lucas W. Remedios, Han Liu, Samuel W. Remedios, Lianrui Zuo, Adam M. Saunders, Shunxing Bao, Yuankai Huo, Alvin C. Powers, John Virostko, Bennett A. LandmanJournal of Medical Imaging 2025 · DOI

TL;DR

This paper investigates where to best combine information from different MRI scans (T2w and T1w) to improve pancreas segmentation. They found that while fusion can help, the optimal "fusion point" within a deep learning model is highly dependent on the specific model architecture, and the performance gains are generally small, especially with imperfectly aligned images.

What Problem Does This Paper Address?

Accurately segmenting organs like the pancreas is crucial for understanding diseases like diabetes. Multimodal imaging, combining data from different scan types (like T2w and T1w MRI), holds promise for better segmentation by providing complementary information. However, a key challenge in medical imaging, particularly for abdominal organs, is that images from different modalities are often not perfectly aligned due to patient movement and the deformable nature of the anatomy. This imperfect registration makes it difficult to effectively fuse the information. The paper tackles the question of where in a deep learning model this fusion should occur – early, late, or somewhere in between – to maximize segmentation performance on these imperfectly registered multimodal images.

Methods

The researchers explored different "fusion points" within two popular deep learning architectures for image segmentation: a basic UNet and the more advanced nnUNet. They trained numerous model variants, each designed to fuse the T2w and T1w MRI data at a specific stage of processing. These stages ranged from early in the network (e.g., concatenating raw images) to later stages (e.g., averaging the model's output predictions, known as logits). They used a state-of-the-art deformable image registration method (deeds) to align the T2w and T1w images before feeding them into the models. The performance of these fusion models was compared against single-modality baselines (using only T2w or only T1w images).

Datasets

The study utilized a dataset of 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MRI images from 163 subjects. These images were acquired from Vanderbilt University Medical Center and were associated with clinical trial data. Pancreas segmentation labels were primarily drawn from the T2w images, with some exceptions due to artifacts. The dataset was de-identified and excluded scans with major motion artifacts.

Key Insights and Ideas

  • Fusion Location is Model-Specific: The "sweet spot" for fusing multimodal MRI data isn't universal. What works best for a basic UNet might not be optimal for a more sophisticated architecture like nnUNet.
  • Gains are Modest: While fusion can offer improvements, the performance gains in terms of segmentation accuracy (measured by Dice score) are generally small, especially when dealing with imperfectly registered images.
  • Imperfect Registration is a Major Hurdle: The inherent challenges of aligning abdominal images significantly impact the effectiveness of fusion. Even advanced registration techniques don't fully overcome this, limiting the potential benefits of combining modalities.
  • Early Fusion Can Be Promising (for some models): For the nnUNet, very early fusion through simple image concatenation showed the most improvement over the baseline.
  • T1w Images are More Challenging: The T1w images, often acquired with different breathing protocols and potentially less clear pancreas delineation, presented a greater challenge for segmentation compared to T2w images. Fusion's impact was more noticeable when the T1w data provided genuinely helpful information.

Key Results

  • Basic UNet: The best fusion approach for the basic UNet was found in the middle of the encoder, achieving a median Dice score improvement of +0.012 compared to the T2w-only baseline.
  • nnUNet: For nnUNet, the best fusion approach was early fusion via naïve image concatenation, yielding a median Dice score improvement of +0.004 over its baseline.
  • Statistical Significance: The improvements observed for the best fusion approaches in both models were statistically significant (p < 0.05) after Bonferroni correction.
  • T1w-Only Baseline: The T1w-only baseline performed worse than the T2w-only baseline, highlighting the difficulty of segmenting the pancreas using T1w images alone.
  • Generalizability: Improvements seen with specific fusion points in the basic UNet did not consistently transfer to the nnUNet, and vice versa, underscoring the model-specific nature of optimal fusion.

Limitations and Future Work

  • Small Gains: The authors acknowledge that the performance gains from fusion are small, suggesting that current methods may not fully exploit the complementary information in imperfectly registered multimodal data.
  • Model Specificity: The lack of generalizable fusion points across different architectures indicates that further innovation is needed to develop more robust fusion strategies.
  • Future Directions: The paper calls for future innovation to better address fusion in scenarios with imperfectly registered image pairs. This could involve exploring more advanced fusion techniques, improved registration methods, or architectures specifically designed to handle misalignment. The authors also excluded hybrid fusion approaches from their study, which could be a direction for future research.

Summary updated

Abstract

Fusion in specific blocks can improve performance, but the best blocks for fusion are model-specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs. The code associated with this project is available here https://github.com/MASILab/influence_of_fusion_on_pancreas_segmentation.
BibTeX
@article{remedios2025influence,
  author = {Lucas W. Remedios and Han Liu and Samuel W. Remedios and Lianrui Zuo and Adam M. Saunders and Shunxing Bao and Yuankai Huo and Alvin C. Powers and John Virostko and Bennett A. Landman},
  title = {{Influence of early through late fusion on pancreas segmentation from imperfectly registered multimodal magnetic resonance imaging}},
  journal = {Journal of Medical Imaging},
  year = {2025},
  doi = {10.1117/1.jmi.12.2.024008},
}
Figure