Abstract
Complementary and redundant relationships inherently exist between multi-modal medical images captured from the same brain. Fusion processes conducted on intermingled representations can cause information distortion and the loss of discriminative modality information. To fully exploit the interdependency between source images for better feature representation and improve the fusion accuracy, we present the multi-modal brain medical image fusion method in a disentangled pipeline under the deep learning framework. A three-branch auto-encoder with two complementary branches and a redundant branch is designed to extract the exclusive modality features and common structure features from input images. Especially, to promote the disentanglement of complement and redundancy, a complementary group lasso penalty is proposed to constrain the extracted feature maps. Then, based on the disentangled representations, different fusion strategies are adopted for complementary features and redundant features, respectively. The experiments demonstrate the superior performance of the proposed fusion method in terms of structure preservation, visual quality, and running efficiency.
Introduction
In clinical neurology, multi-modal imaging is essential. MRI provides high-resolution soft tissue contrast, CT offers skeletal detail, and PET reveals metabolic activity. Fusing these modalities into a single image allows clinicians to synthesize diverse diagnostic information. However, traditional fusion methods often struggle with "intermingled representations," where unique modality-specific details (complementary features) and shared structural information (redundant features) are mixed. This leads to artifacts like blurring, ghosting, or the loss of critical diagnostic details.
Methods
The proposed framework addresses these issues through a disentangled representation pipeline. The core architecture is a three-branch auto-encoder designed to separate features into distinct categories.
Three-Branch Auto-Encoder
The architecture consists of:
- Complementary Encoders ($En_{C1}, En_{C2}$): These branches extract features unique to each source modality (e.g., the specific metabolic hotspots in PET or the bone density in CT).
- Redundant Encoder ($En_R$): This shared branch extracts the common structural information present in both images, such as the overall brain geometry.
Complementary Group Lasso Penalty
To ensure effective disentanglement, the authors introduce a novel complementary group lasso penalty. This constraint acts as a mathematical "traffic cop," penalizing the complementary branches if they attempt to encode redundant information. This forces the model to isolate unique details in the $C$ feature maps and shared structures in the $R$ feature maps.
Fusion Strategy
Once disentangled, the features are fused using specialized strategies:
- Redundant features ($R_1, R_2$): Combined via averaging to maintain structural consistency.
- Complementary features ($C_1, C_2$): Combined using "max-selection" or L1-norm weighting to preserve the most salient unique details.
The fused feature maps ($C_f, R_f$) are then passed through a decoder ($De_S$) to reconstruct the final fused image $I_f$.
Experimental Results
The model was evaluated on the Harvard Medical Dataset, covering MRI-CT (180 pairs) and MRI-PET (260 pairs) tasks.
Performance and Efficiency
The proposed method outperformed six state-of-the-art models (including EMFusion and DDcGAN) across 11 objective metrics. Key findings include:
- Visual Quality: Fused images exhibited sharper edges and superior contrast, effectively merging functional PET data with high-resolution MRI structures without blurring.
- Quantitative Metrics: The method achieved top scores in $Q_G$ (gradient preservation) and $Q_{CB}$ (visual perception), indicating high fidelity to the source images.
- Computational Speed: The system is remarkably efficient, processing a 256×256 image in just 0.037 seconds. This is significantly faster than competing GAN-based or optimization-based methods, which can take several seconds per image.
Conclusion
By explicitly disentangling complementary and redundant features using a group lasso penalty, this research provides a robust framework for high-quality medical image fusion. While future work will focus on better integrating color (chrominance) channels directly into the disentanglement process, the current model offers a significant leap in both visual accuracy and real-time processing capability for clinical applications.
Summary updated