Influence of Early through Late Fusion on Pancreas Segmentation from Imperfectly Registered Multimodal MRI

Lucas W. Remedios, Han Liu, Samuel W. Remedios, Lianrui Zuo, Adam M. Saunders, Shunxing Bao, Yuankai Huo, Alvin C. Powers, John Virostko, Bennett A. LandmanarXiv 2024 · DOI · arXiv

Abstract

Pancreas segmentation in abdominal MRI is challenging due to the organ's variable morphology and poor contrast boundaries. While multimodal imaging (e.g., T1-weighted and T2-weighted) offers complementary information, imperfect spatial registration between modalities often limits performance. This paper investigates the optimal "fusion point"—the stage at which a deep learning model combines multimodal data—across different architectures. We demonstrate that while multimodal fusion provides a statistically significant boost in Dice scores, the ideal location to merge data is architecture-specific and influenced by the quality of image registration.

Introduction

The pancreas is notoriously difficult to segment in medical imaging due to its lack of distinct boundaries and high inter-patient variability. Utilizing multiple imaging modalities, such as T1-weighted (T1w) and T2-weighted (T2w) MRI, should theoretically improve segmentation by providing richer anatomical context. However, abdominal imaging faces significant hurdles: the region is highly deformable, and patient respiration during scans leads to misalignments.

Even with state-of-the-art deformable registration techniques, T1 and T2 images are rarely perfectly aligned. This "imperfect registration" creates a fundamental architectural question: where should we combine the data? Should we merge raw images at the input (early fusion), combine feature maps within the network (intermediate fusion), or average final predictions (late fusion)?

T1 and T2 MRI registration challenges and ground truth pancreas visualization.

Methods

The researchers systematically evaluated fusion strategies using two popular deep learning architectures: a Basic UNet (from the MONAI framework) and the more robust nnUNet.

Dataset and Registration

The study utilized 353 pairs of T2w and T1w abdominal MR images from 163 subjects, including healthy controls and patients with various forms of diabetes. Ground-truth labels were primarily established based on T2w images. To align the modalities, the "deeds" deformable registration technique was employed to map T1w images onto the T2w space.

Fusion Strategies

The authors tested 13 variants for the Basic UNet and 17 for the nnUNet, covering the entire pipeline:

  • Early Fusion: Concatenating T1w and T2w images before model entry.
  • Intermediate Fusion: Merging feature maps at various stages (encoder, bottleneck, or decoder) via concatenation.
  • Late Fusion: Averaging the final probability maps (logits) from two independent models.
Architectural overview of Basic UNet and nnUNet showing various fusion points from early through late stages.

Results

Performance was measured using the Dice score (0.0 to 1.0). The T2w-only baselines achieved scores of 0.73 for the Basic UNet and 0.80 for the nnUNet.

  • Optimal Fusion Points: There was no "silver bullet." For the Basic UNet, fusion in the middle of the encoder yielded the best results, improving the Dice score by 0.0125. Conversely, for the nnUNet, simple early concatenation was most effective, providing a smaller but significant gain of 0.0021.
  • Impact of Pancreas Size: Fusion provided the most substantial benefits for smaller pancreases, which are typically harder to segment. This suggests multimodal data is most valuable when the primary signal (T2w) is weak or ambiguous.
  • Architecture Sensitivity: Improvements observed in one model did not necessarily translate to the other, indicating that fusion strategies must be tailored to the specific neural network architecture.

Discussion and Conclusion

The study highlights that the "art of design" remains critical because current registration techniques cannot perfectly align abdominal organs. The choice of fusion point essentially determines how much registration "noise" the model can tolerate at different levels of abstraction. Qualitative analysis confirmed that while T1w data can clarify boundaries, poorly aligned images can introduce confusion and degrade performance.

Future work should focus on "registration-aware" fusion layers that can dynamically adjust to local misalignments, rather than relying on static fusion points. While the absolute gains in this study were modest, they underscore the potential of multimodal fusion to improve clinical segmentation tasks, particularly in challenging cases like diabetic patients with reduced pancreatic volume.

Summary updated

Abstract

Multimodal fusion promises better pancreas segmentation. However, where to perform fusion in models is still an open question. It is unclear if there is a best location to fuse information when analyzing pairs of imperfectly aligned images. Two main alignment challenges in this pancreas segmentation study are 1) the pancreas is deformable and 2) breathing deforms the abdomen. Even after image registration, relevant deformations are often not corrected. We examine how early through late fusion impacts pancreas segmentation. We used 353 pairs of T2-weighted (T2w) and T1-weighted (T1w) abdominal MR images from 163 subjects with accompanying pancreas labels. We used image registration (deeds) to align the image pairs. We trained a collection of basic UNets with different fusion points, spanning from early to late, to assess how early through late fusion influenced segmentation performance on imperfectly aligned images. We assessed generalization of fusion points on nnUNet. The single-modality T2w baseline using a basic UNet model had a Dice score of 0.73, while the same baseline on the nnUNet model achieved 0.80. For the basic UNet, the best fusion approach occurred in the middle of the encoder (early/mid fusion), which led to a statistically significant improvement of 0.0125 on Dice score compared to the baseline. For the nnUNet, the best fusion approach was naïve image concatenation before the model (early fusion), which resulted in a statistically significant Dice score increase of 0.0021 compared to baseline. Fusion in specific blocks can improve performance, but the best blocks for fusion are model specific, and the gains are small. In imperfectly registered datasets, fusion is a nuanced problem, with the art of design remaining vital for uncovering potential insights. Future innovation is needed to better address fusion in cases of imperfect alignment of abdominal image pairs.
BibTeX
@misc{remedios2024influence,
  author = {Lucas W. Remedios and Han Liu and Samuel W. Remedios and Lianrui Zuo and Adam M. Saunders and Shunxing Bao and Yuankai Huo and Alvin C. Powers and John Virostko and Bennett A. Landman},
  title = {{Influence of Early through Late Fusion on Pancreas Segmentation from Imperfectly Registered Multimodal MRI}},
  year = {2024},
  eprint = {2409.04563},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.04563},
}
Figure