Abstract
Pancreas segmentation in abdominal MRI is challenging due to the organ's variable morphology and poor contrast boundaries. While multimodal imaging (e.g., T1-weighted and T2-weighted) offers complementary information, imperfect spatial registration between modalities often limits performance. This paper investigates the optimal "fusion point"—the stage at which a deep learning model combines multimodal data—across different architectures. We demonstrate that while multimodal fusion provides a statistically significant boost in Dice scores, the ideal location to merge data is architecture-specific and influenced by the quality of image registration.
Introduction
The pancreas is notoriously difficult to segment in medical imaging due to its lack of distinct boundaries and high inter-patient variability. Utilizing multiple imaging modalities, such as T1-weighted (T1w) and T2-weighted (T2w) MRI, should theoretically improve segmentation by providing richer anatomical context. However, abdominal imaging faces significant hurdles: the region is highly deformable, and patient respiration during scans leads to misalignments.
Even with state-of-the-art deformable registration techniques, T1 and T2 images are rarely perfectly aligned. This "imperfect registration" creates a fundamental architectural question: where should we combine the data? Should we merge raw images at the input (early fusion), combine feature maps within the network (intermediate fusion), or average final predictions (late fusion)?
Methods
The researchers systematically evaluated fusion strategies using two popular deep learning architectures: a Basic UNet (from the MONAI framework) and the more robust nnUNet.
Dataset and Registration
The study utilized 353 pairs of T2w and T1w abdominal MR images from 163 subjects, including healthy controls and patients with various forms of diabetes. Ground-truth labels were primarily established based on T2w images. To align the modalities, the "deeds" deformable registration technique was employed to map T1w images onto the T2w space.
Fusion Strategies
The authors tested 13 variants for the Basic UNet and 17 for the nnUNet, covering the entire pipeline:
- Early Fusion: Concatenating T1w and T2w images before model entry.
- Intermediate Fusion: Merging feature maps at various stages (encoder, bottleneck, or decoder) via concatenation.
- Late Fusion: Averaging the final probability maps (logits) from two independent models.
Results
Performance was measured using the Dice score (0.0 to 1.0). The T2w-only baselines achieved scores of 0.73 for the Basic UNet and 0.80 for the nnUNet.
- Optimal Fusion Points: There was no "silver bullet." For the Basic UNet, fusion in the middle of the encoder yielded the best results, improving the Dice score by 0.0125. Conversely, for the nnUNet, simple early concatenation was most effective, providing a smaller but significant gain of 0.0021.
- Impact of Pancreas Size: Fusion provided the most substantial benefits for smaller pancreases, which are typically harder to segment. This suggests multimodal data is most valuable when the primary signal (T2w) is weak or ambiguous.
- Architecture Sensitivity: Improvements observed in one model did not necessarily translate to the other, indicating that fusion strategies must be tailored to the specific neural network architecture.
Discussion and Conclusion
The study highlights that the "art of design" remains critical because current registration techniques cannot perfectly align abdominal organs. The choice of fusion point essentially determines how much registration "noise" the model can tolerate at different levels of abstraction. Qualitative analysis confirmed that while T1w data can clarify boundaries, poorly aligned images can introduce confusion and degrade performance.
Future work should focus on "registration-aware" fusion layers that can dynamically adjust to local misalignments, rather than relying on static fusion points. While the absolute gains in this study were modest, they underscore the potential of multimodal fusion to improve clinical segmentation tasks, particularly in challenging cases like diabetic patients with reduced pancreatic volume.
Summary updated