Multimodal-Boost: Multimodal Medical Image Super-Resolution using Multi-Attention Network with Wavelet Transform

Farah Deeba, Fayaz Ali Dharejo, Muhammad Zawish, Yuanchun Zhou, Kapal Dev, Sunder Ali Khowaja, Nawab Muhammad Faseeh Qureshi

TL;DR

This paper introduces "Multimodal-Boost," a novel deep learning framework for enhancing the resolution of medical images. By combining wavelet transforms, multi-attention generative adversarial networks (GANs), and transfer learning, it addresses the critical need for high-quality medical images for accurate diagnosis, outperforming existing methods in both detail and modality adaptability.

What Problem Does This Paper Address?

High-resolution medical images are crucial for accurate diagnosis, but factors like imaging modalities and acquisition speed often lead to degraded spatial resolution. This loss of detail can hinder interpretation tasks like segmentation and diagnosis. While deep learning, particularly GANs, has shown promise in single image super-resolution (SISR), existing methods often produce outputs with poor texture details and blurred edges, which is especially problematic for medical imaging. Furthermore, many models are modality-specific, requiring significant modifications to work with different types of medical scans. This paper aims to tackle these limitations by developing a versatile SISR model that can generate high-fidelity, detailed super-resolved images from low-resolution medical scans across various modalities.

Methods

The core of the "Multimodal-Boost" approach lies in a three-part methodology:

  • Wavelet Transform (WT) Decomposition: The low-resolution (LR) input image is first decomposed into four subbands (LL, LH, HL, HH) using a 2D discrete wavelet transform (DWT). This process effectively separates the image into different frequency components, allowing the model to focus on reconstructing high-frequency details.
  • Multi-Attention GAN for Prediction: These wavelet subbands are then fed into a novel Generative Adversarial Network (GAN) that incorporates "multi-attention" and "upsample-attention" blocks. These attention mechanisms are designed to learn and predict the high-frequency components needed to reconstruct the high-resolution (HR) wavelet subbands. The GAN architecture includes both a generator and a discriminator, trained to produce realistic HR images.
  • Reconstruction and Perceptual Loss: The predicted HR wavelet subbands are then combined using a 2D inverse discrete wavelet transform (IDWT) to generate the final super-resolved image. To further enhance the perceptual quality and detail, the model utilizes a VGG-based perceptual loss function, which encourages the generated image to match the perceptual characteristics of the ground truth HR image, rather than just pixel-wise accuracy.
  • Transfer Learning for Multimodality: A key innovation is the use of transfer learning. The model is initially pre-trained on a large, general-purpose dataset (DIV2K) and then fine-tuned on specific medical image datasets. This allows the model to adapt to different medical modalities (e.g., CT, MRI, X-ray) without needing to be trained from scratch for each, significantly reducing the cost and effort of adapting the SR solution to new imaging types.

Datasets

  • Training (Pre-training): DIV2K dataset, a high-quality dataset with 2K resolution images, used for initial model training.
  • Training/Fine-tuning: Shenzhen Hospital datasets, containing 662 X-ray images.
  • Testing/Evaluation:
    • CT images (Teeth, Knee) from Shenzhen Hospital datasets.
  • MRI brain scans from the Calgary Campinas repository.
  • Additional modalities tested include Montgomery County X-ray and "Teeth" and "knee" images.

Key Insights and Ideas

  • Wavelets as Feature Extractors: Decomposing images into wavelet subbands proves effective for isolating and reconstructing high-frequency details, which are often lost in low-resolution medical images.
  • Attention for Detail: Multi-attention mechanisms within the GAN are crucial for capturing and leveraging subtle, long-range dependencies in the image data, leading to better texture and edge reconstruction.
  • Perceptual Loss for Realism: Employing a VGG-based perceptual loss function, rather than solely relying on traditional metrics like PSNR/SSIM, significantly improves the visual quality and realism of the super-resolved medical images.
  • Transfer Learning as a Unifier: The successful application of transfer learning across diverse medical modalities highlights its power in creating a single, adaptable SR model for various diagnostic imaging needs, reducing the burden of modality-specific development.

Key Results

  • Quantitative Performance: The proposed "Multimodal-Boost" method achieves state-of-the-art results, outperforming methods like SRCNN, EDSR, Meta-SR, CIRCLE-GAN, and SR-ILLNN in terms of PSNR and SSIM across multiple medical modalities (CT, MRI, X-ray). For instance, it achieves 1.54-4.89 higher PSNR compared to recent methods.
  • Qualitative Performance: Visual results demonstrate that Multimodal-Boost generates super-resolved images with significantly sharper edges and richer texture details compared to other methods, which often produce blurred or over-smoothed outputs. This is particularly evident in magnified regions of the images.
  • Efficiency: While the model has a moderate number of parameters, it offers a good balance between performance and computational cost. The paper also notes that WGAN-based losses converge faster than non-WGAN-based losses, and combining WGAN with perceptual loss leads to shorter distances and greater convergence.

Limitations and Future Work

  • Model Compression: The authors acknowledge that further work on model compression could be beneficial to reduce computational load and improve efficiency.
  • Target Task Performance: While the SR quality is improved, the paper suggests future work to study how this enhanced SR approach affects downstream tasks like disease detection and classification.
  • Training Time: The proposed method takes longer to train than some other state-of-the-art methods like Meta-SR.
  • Generalization: While transfer learning is used, further exploration into optimizing the transfer learning process for even broader modality coverage and robustness could be valuable.

Summary updated

Abstract

Deep learning based single image super resolution (SISR) algorithms has revolutionized the overall diagnosis framework by continually improving the architectural components and training strategies associated with convolutional neural networks (CNN) on low-resolution images. However, existing work lacks in two ways: i) the SR output produced exhibits poor texture details, and often produce blurred edges, ii) most of the models have been developed for a single modality, hence, require modification to adapt to a new one. This work addresses (i) by proposing generative adversarial network (GAN) with deep multi-attention modules to learn high-frequency information from low-frequency data. Existing approaches based on the GAN have yielded good SR results; however, the texture details of their SR output have been experimentally confirmed to be deficient for medical images particularly. The integration of wavelet transform (WT) and GANs in our proposed SR model addresses the aforementioned limitation concerning textons. While the WT divides the LR image into multiple frequency bands, the transferred GAN uses multi-attention and upsample blocks to predict high-frequency components. Additionally, we present a learning method for training domain-specific classifiers as perceptual loss functions. Using a combination of multi-attention GAN loss and a perceptual loss function results in an efficient and reliable performance. Applying the same model for medical images from diverse modalities is challenging, our work addresses (ii) by training and performing on several modalities via transfer learning. Using two medical datasets, we validate our proposed SR network against existing state-of-the-art approaches and achieve promising results in terms of SSIM and PSNR.
BibTeX
@article{deeba2021multimodal,
  author = {Farah Deeba and Fayaz Ali Dharejo and Muhammad Zawish and Yuanchun Zhou and Kapal Dev and Sunder Ali Khowaja and Nawab Muhammad Faseeh Qureshi},
  title = {{Multimodal-Boost: Multimodal Medical Image Super-Resolution using Multi-Attention Network with Wavelet Transform}},
  journal = {IEEE ACM Trans. Comput. Biol. Bioinform.},
  year = {2021},
  url = {https://openreview.net/forum?id=NagUplXUkfu},
}
Figure