Rethinking Multi-Contrast MRI Super-Resolution: Rectangle-Window Cross-Attention Transformer and Arbitrary-Scale Upsampling

Guangyuan Li, Lei Zhao, Jiakai Sun, Zehua Lan, Zhanjie Zhang, Jiafu Chen, Zhijie Lin, Huaizhong Lin, Wei Xing

Abstract

Magnetic Resonance Imaging (MRI) often involves a trade-off between image quality and acquisition time. While multi-contrast super-resolution (SR) techniques leverage high-resolution reference scans to sharpen lower-quality target scans, existing methods suffer from rigid scaling limitations and inefficient square-window processing. This paper introduces McASSR, a framework utilizing a novel Rectangle-Window Cross-Attention Transformer (RCT) and Reference-Aware Implicit Attention (RIA). McASSR enables high-quality SR at any magnification level, capturing long-range anatomical dependencies more effectively than traditional methods.

The Challenge of MRI Super-Resolution

In clinical settings, high-resolution (HR) images are essential for diagnosis but time-consuming to acquire, often leading to patient discomfort and motion artifacts. Super-resolution—using AI to upscale low-resolution (LR) scans—offers a solution. Multi-contrast SR specifically uses a fast-acquired reference scan (e.g., T1-weighted) to help reconstruct a slower target scan (e.g., T2-weighted).

However, current state-of-the-art models face two primary hurdles:

  1. Rigid Scaling: Most models are restricted to fixed integer scales (e.g., 2x, 4x). Non-integer or arbitrary scales require retraining, which is clinically impractical.
  2. Square-Window Limitations: Standard transformers use square windows (e.g., 8x8) that fail to capture the elongated anatomical structures and global symmetries inherent in human organs like the brain or knee.

The McASSR Framework

To address these issues, the authors developed McASSR (Multi-Contrast Arbitrary-Scale Super-Resolution). The architecture integrates two primary innovations to handle complex medical textures and flexible zoom requirements.

The McASSR architecture overview, featuring the Rectangle-Window Cross-Attention Transformer and the Reference-Aware Implicit Attention module for arbitrary-scale upsampling.

Innovation 1: Rectangle-Window Cross-Attention Transformer (RCT)

Human anatomy is rarely square. Structures like the cortex or long bones are better represented by elongated windows. The RCT utilizes horizontal and vertical rectangle windows, allowing the model to "see" further across the image without increasing computational costs. This approach establishes superior long-range dependencies, which are crucial for maintaining structural integrity in medical imaging.

Innovation 2: Reference-Aware Implicit Attention (RIA)

For arbitrary scaling, McASSR moves away from traditional discrete upsampling (like bicubic interpolation) toward Implicit Neural Representation (INR). By treating the image as a continuous function, any coordinate $(x, y)$ can be mapped to a pixel value. The RIA module specifically queries the high-resolution reference image for high-frequency textures, injecting this detail into the target image during the upscaling process. This ensures that fine details are preserved even at non-integer magnification levels.

Experimental Results

The model was evaluated on the FastMRI (knee) and BraTs (brain) datasets, alongside two in-house clinical datasets. McASSR consistently outperformed state-of-the-art methods across all metrics.

  • Quantitative Performance: On the FastMRI dataset at 4x scale, McASSR achieved a PSNR of 32.52 dB, significantly outperforming other arbitrary-scale methods like LTE (31.19 dB).
  • Out-of-Scale Generalization: When trained on 4x scales but tested at 6x or 8x, McASSR maintained sharper edges and fewer artifacts than its competitors.
Qualitative comparison of McASSR against LTE* and Target HR across various scales (4.0x, 5.6x, and 7.2x) for brain and knee scans.

Clinical Implications and Future Work

By shifting SR from a discrete image-processing task to a continuous coordinate-mapping task, McASSR provides radiologists with a "future-proof" tool for any required zoom level. The efficiency of the model—with inference times around 35ms—makes it a strong candidate for real-time clinical integration.

A current limitation remains image registration; the reference and target scans must be perfectly aligned for optimal fusion. Future research will focus on multi-task frameworks that perform registration and super-resolution simultaneously, as well as extending the architecture to handle 3D volumetric MRI data.

Summary updated

Abstract

Recently, several methods have explored the potential of multi-contrast magnetic resonance imaging (MRI) super-resolution (SR) and obtain results superior to single-contrast SR methods. However, existing approaches still have two shortcomings: (1) They can only address fixed integer upsampling scales, such as 2×, 3×, and 4×, which require training and storing the corresponding model separately for each upsampling scale in clinic. (2) They lack direct interaction among different windows as they adopt the square window (e.g., 8×8) transformer network architecture, which results in inadequate modelling of longer-range dependencies. Moreover, the relationship between reference images and target images is not fully mined. To address these issues, we develop a novel network for multi-contrast MRI arbitrary-scale SR, dubbed as McASSR. Specifically, we design a rectangle-window cross-attention transformer to establish longer-range dependencies in MR images without increasing computational complexity and fully use reference information. Besides, we propose the reference-aware implicit attention as an upsampling module, achieving arbitrary-scale super-resolution via implicit neural representation, further fusing supplementary information of the reference image. Extensive and comprehensive experiments on both public and clinical datasets show that our McASSR yields superior performance over SOTA methods, demonstrating its great potential to be applied in clinical practice. Code will be available at https://github.com/GuangYuanKK/McASSR.
BibTeX
@inproceedings{li2023rethinking,
  author = {Guangyuan Li and Lei Zhao and Jiakai Sun and Zehua Lan and Zhanjie Zhang and Jiafu Chen and Zhijie Lin and Huaizhong Lin and Wei Xing},
  title = {{Rethinking Multi-Contrast MRI Super-Resolution: Rectangle-Window Cross-Attention Transformer and Arbitrary-Scale Upsampling}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01941},
}
Figure