A Disentangled Representation Based Brain Image Fusion via Group Lasso Penalty

Anqi Wang, Xiaoqing Luo, Zhancheng Zhang, Xiao‐Jun WuFrontiers in Neuroscience 2022 · DOI

Abstract

Complementary and redundant relationships inherently exist between multi-modal medical images captured from the same brain. Fusion processes conducted on intermingled representations can cause information distortion and the loss of discriminative modality information. To fully exploit the interdependency between source images for better feature representation and improve the fusion accuracy, we present the multi-modal brain medical image fusion method in a disentangled pipeline under the deep learning framework. A three-branch auto-encoder with two complementary branches and a redundant branch is designed to extract the exclusive modality features and common structure features from input images. Especially, to promote the disentanglement of complement and redundancy, a complementary group lasso penalty is proposed to constrain the extracted feature maps. Then, based on the disentangled representations, different fusion strategies are adopted for complementary features and redundant features, respectively. The experiments demonstrate the superior performance of the proposed fusion method in terms of structure preservation, visual quality, and running efficiency.

Introduction

In clinical neurology, multi-modal imaging is essential. MRI provides high-resolution soft tissue contrast, CT offers skeletal detail, and PET reveals metabolic activity. Fusing these modalities into a single image allows clinicians to synthesize diverse diagnostic information. However, traditional fusion methods often struggle with "intermingled representations," where unique modality-specific details (complementary features) and shared structural information (redundant features) are mixed. This leads to artifacts like blurring, ghosting, or the loss of critical diagnostic details.

Methods

The proposed framework addresses these issues through a disentangled representation pipeline. The core architecture is a three-branch auto-encoder designed to separate features into distinct categories.

Three-Branch Auto-Encoder

The architecture consists of:

  1. Complementary Encoders ($En_{C1}, En_{C2}$): These branches extract features unique to each source modality (e.g., the specific metabolic hotspots in PET or the bone density in CT).
  2. Redundant Encoder ($En_R$): This shared branch extracts the common structural information present in both images, such as the overall brain geometry.
The overall architecture of the proposed disentangled fusion framework (A) and the subsequent fusion process (B).

Complementary Group Lasso Penalty

To ensure effective disentanglement, the authors introduce a novel complementary group lasso penalty. This constraint acts as a mathematical "traffic cop," penalizing the complementary branches if they attempt to encode redundant information. This forces the model to isolate unique details in the $C$ feature maps and shared structures in the $R$ feature maps.

Detail of the Complementary Group Lasso penalty applied to feature vectors at each (x,y) position.

Fusion Strategy

Once disentangled, the features are fused using specialized strategies:

  • Redundant features ($R_1, R_2$): Combined via averaging to maintain structural consistency.
  • Complementary features ($C_1, C_2$): Combined using "max-selection" or L1-norm weighting to preserve the most salient unique details.

The fused feature maps ($C_f, R_f$) are then passed through a decoder ($De_S$) to reconstruct the final fused image $I_f$.

Experimental Results

The model was evaluated on the Harvard Medical Dataset, covering MRI-CT (180 pairs) and MRI-PET (260 pairs) tasks.

Performance and Efficiency

The proposed method outperformed six state-of-the-art models (including EMFusion and DDcGAN) across 11 objective metrics. Key findings include:

  • Visual Quality: Fused images exhibited sharper edges and superior contrast, effectively merging functional PET data with high-resolution MRI structures without blurring.
  • Quantitative Metrics: The method achieved top scores in $Q_G$ (gradient preservation) and $Q_{CB}$ (visual perception), indicating high fidelity to the source images.
  • Computational Speed: The system is remarkably efficient, processing a 256×256 image in just 0.037 seconds. This is significantly faster than competing GAN-based or optimization-based methods, which can take several seconds per image.

Conclusion

By explicitly disentangling complementary and redundant features using a group lasso penalty, this research provides a robust framework for high-quality medical image fusion. While future work will focus on better integrating color (chrominance) channels directly into the disentanglement process, the current model offers a significant leap in both visual accuracy and real-time processing capability for clinical applications.

Summary updated

Abstract

Complementary and redundant relationships inherently exist between multi-modal medical images captured from the same brain. Fusion processes conducted on intermingled representations can cause information distortion and the loss of discriminative modality information. To fully exploit the interdependency between source images for better feature representation and improve the fusion accuracy, we present the multi-modal brain medical image fusion method in a disentangled pipeline under the deep learning framework. A three-branch auto-encoder with two complementary branches and a redundant branch is designed to extract the exclusive modality features and common structure features from input images. Especially, to promote the disentanglement of complement and redundancy, a complementary group lasso penalty is proposed to constrain the extracted feature maps. Then, based on the disentangled representations, different fusion strategies are adopted for complementary features and redundant features, respectively. The experiments demonstrate the superior performance of the proposed fusion method in terms of structure preservation, visual quality, and running efficiency.
BibTeX
@inproceedings{wang2022disentangled,
  author = {Anqi Wang and Xiaoqing Luo and Zhancheng Zhang and Xiao‐Jun Wu},
  title = {{A Disentangled Representation Based Brain Image Fusion via Group Lasso Penalty}},
  booktitle = {Frontiers in Neuroscience},
  year = {2022},
  doi = {10.3389/fnins.2022.937861},
}
Figure