An Attention-based Multi-Scale Feature Learning Network for Multimodal Medical Image Fusion

Meng Zhou, Xiaolan Xu, Yuxuan ZhangarXiv 2022 · OpenReview

Abstract

Medical images play an important role in clinical applications. Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. This technique will prevent radiologists from switching back and forth between different images and save significant time in the diagnostic process. In this paper, we introduce a novel Dilated Residual Attention Network (DILRAN) for the medical image fusion task. Our network is capable of extracting multi-scale deep semantic features. Furthermore, we propose a novel fixed fusion strategy termed Softmax-based weighted strategy based on the Softmax weights and matrix nuclear norm. Extensive experiments show our proposed network and fusion strategy exceed state-of-the-art performance compared with reference image fusion methods on four commonly used fusion metrics.

Introduction

In clinical diagnosis, physicians often rely on multiple imaging modalities to get a complete picture of a patient's health. For example, MRI provides excellent soft-tissue detail, while CT scans are superior for highlighting bone structures. Manually comparing these separate images is time-consuming and mentally taxing for radiologists. Multimodal Medical Image Fusion (MMIF) aims to solve this by synthesizing complementary information into one high-detail image.

Existing deep learning methods for MMIF often struggle with information loss during standard downsampling (pooling) and suboptimal fusion strategies like simple pixel averaging, which can result in low-contrast images. To address these challenges, we developed DILRAN (Dilated Residual Attention Network), a framework that preserves sharp edges and complex textures through advanced attention mechanisms and dilated convolutions.

Methods

The proposed end-to-end pipeline consists of three main components: a feature extractor, a fusion module, and a reconstruction module. The system processes input images (such as CT and MRI) through parallel feature extraction branches before combining them.

The overall pipeline of the proposed medical image fusion framework, showing the feature extraction, fusion, and reconstruction stages for CT and MRI inputs.

DILRAN Feature Extractor

The "brain" of the operation is the DILRAN module. Unlike traditional networks that use pooling, DILRAN utilizes Dilated Convolutions (with rates of 1, 3, and 5) to expand the receptive field and capture multi-scale features without losing spatial resolution. It integrates Residual Attention and Pyramid Attention modules to help the network focus on the most relevant semantic features, ensuring that critical diagnostic details are maintained.

Detailed architecture of the DILRAN module, illustrating the arrangement of convolution blocks (CB) and attention mechanisms used to extract deep features.

Softmax Feature Weighted Strategy

Instead of simple mathematical operations to combine images, we introduce a strategy based on the matrix nuclear norm and Softmax weights. This allows the system to adaptively decide how much "weight" to give to a specific pixel from the MRI versus the CT scan based on the information density in that area. This fusion strategy is "fixed" (non-trainable during inference), making it computationally efficient and stable while remaining highly adaptive to input features.

Hybrid Loss Function

To ensure high-quality results, the model is trained using a hybrid loss function that combines:

  • Mean Squared Error (MSE): For general pixel-wise accuracy.
  • Gradient Loss: To keep edges sharp and clear.
  • Perceptual Loss: Using a pre-trained VGG16 network to ensure the fused image is semantically correct and visually natural to the human eye.

Experimental Results

The framework was evaluated using the "Harvard Whole Brain Atlas" dataset, focusing on 184 MRI-CT pairs. Images were resized to 256x256 and normalized for training.

Our method was compared against several baseline methods, including Zero-shot learning, MSDRA, and MSRPAN. DILRAN achieved the highest scores across several quantitative metrics:

  • PSNR (Peak Signal-to-Noise Ratio): Hit 16.413, significantly outperforming the Zero-shot method (13.525).
  • FMI-pixel (Feature Mutual Information): Demonstrated superior information retention.
  • FSIM (Feature Similarity Index) and Entropy: Confirmed high structural similarity and rich information content.

Qualitatively, the fused images showed much clearer boundaries between different brain tissues and better intensity contrast compared to older methods, which often produced blurry or "washed out" results.

Conclusion and Future Work

DILRAN represents a significant step forward in medical image fusion by combining dilated convolutions with synergistic attention mechanisms. While the model performs exceptionally well, future work will focus on addressing "corner cases" where one modality's features are extremely faint. We also plan to optimize the network for 3-channel color images (such as PET/SPECT) and further reduce potential artifacts to maintain the highest possible fidelity for clinical use.

Summary updated

Abstract

Medical images play an important role in clinical applications. Multimodal medical images could provide rich information about patients for physicians to diagnose. The image fusion technique is able to synthesize complementary information from multimodal images into a single image. This technique will prevent radiologists switch back and forth between different images and save lots of time in the diagnostic process. In this paper, we introduce a novel Dilated Residual Attention Network for the medical image fusion task. Our network is capable to extract multi-scale deep semantic features. Furthermore, we propose a novel fixed fusion strategy termed Softmax-based weighted strategy based on the Softmax weights and matrix nuclear norm. Extensive experiments show our proposed network and fusion strategy exceed the state-of-the-art performance compared with reference image fusion methods on four commonly used fusion metrics.
BibTeX
@misc{zhou2022attention,
  author = {Meng Zhou and Xiaolan Xu and Yuxuan Zhang},
  title = {{An Attention-based Multi-Scale Feature Learning Network for Multimodal Medical Image Fusion}},
  year = {2022},
  eprint = {2212.04661},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2212.04661},
}
Figure