Inter-slice Super-resolution of Magnetic Resonance Images by Pre-training and Self-supervised Fine-tuning

Xin Wang, Zhiyun Song, Yitao Zhu, Sheng Wang, Lichi Zhang, Dinggang Shen, Qian Wang

Abstract

In clinical practice, 2D magnetic resonance (MR) sequences are widely adopted. While individual 2D slices can be stacked to form a 3D volume, the relatively large slice spacing can pose challenges for both image visualization and subsequent analysis tasks, which often require isotropic voxel spacing. To reduce slice spacing, deep-learning-based super-resolution techniques are widely investigated. However, most current solutions require a substantial number of paired high-resolution and low-resolution images for supervised training, which are typically unavailable in real-world scenarios. In this work, we propose a self-supervised super-resolution framework for inter-slice super-resolution of MR images. Our framework is first featured by pre-training on video dataset, as temporal correlation of videos is found beneficial for modeling the spatial relation among MR slices. Then, we use public high-quality MR dataset to fine-tune our pre-trained model, for enhancing awareness of our model to medical data. Finally, given a target dataset at hand, we utilize self-supervised fine-tuning to further ensure our model works well with user-specific super-resolution tasks. The proposed method demonstrates superior performance compared to other self-supervised methods and also holds the potential to benefit various downstream applications.

Introduction

In clinical settings, 2D Magnetic Resonance (MR) sequences are the standard due to their rapid acquisition times. However, these scans frequently suffer from the "thick-slice" problem: while the resolution within a single 2D slice is high, the gap between slices is relatively large. This results in anisotropic voxels—essentially tall, rectangular prisms rather than uniform cubes—which complicates 3D anatomical visualization and hinders automated AI-based diagnostic tools.

Super-resolution (SR) techniques aim to "fill in the gaps" between these slices. However, most deep learning models require thousands of paired low-resolution (LR) and high-resolution (HR) images for training. In clinical reality, HR versions of specific patient scans are rarely available. While existing self-supervised methods attempt to circumvent this, they often produce blurry results or fail to generalize across different scanners. This paper presents a three-stage framework designed to achieve near-supervised performance without the need for paired clinical ground truth.

Methodology

The proposed pipeline bridges the gap between general motion patterns and specific medical anatomy through three distinct stages:

1. Pre-training on Video (Temporal-to-Spatial Transfer)

The researchers identified a mathematical parallel between a sequence of video frames and a sequence of MR slices. They utilized a Spatially-Aware Implicit Neural Representation (SA-INR) model, pre-trained on the REDS-VTSR video dataset. By learning to interpolate missing frames in natural videos, the model develops a foundational understanding of how to model spatial correlations and continuity.

2. Medical Domain Adaptation

To transition from natural scenes to clinical data, the model is fine-tuned on the Osteoarthritis Initiative (OAI) dataset, which contains 350 high-quality, thin-slice 3D MR volumes. This stage allows the model to learn the specific textures, contrasts, and anatomical structures inherent in medical imaging, effectively "teaching" it what a human knee or bone should look like.

3. Subject-Specific Self-supervised Fine-tuning (SSF)

To account for variations in scanner settings and individual patient anatomy, the model performs a final "tune-up" on the target scan itself. By downsampling the patient's own LR scan to create a proxy training set, the model adapts to the specific noise profile and characteristics of that individual volume. This process is efficient, taking approximately one minute on a modern GPU.

Experimental Results

The framework was evaluated using the SKI10 (Segmentation of Knee Images 2010) dataset, consisting of 150 clinical-grade knee scans.

Quantitative Performance

The proposed method achieved a Peak Signal-to-Noise Ratio (PSNR) of 30.88 dB. For comparison, a fully supervised model—which represents the theoretical upper bound but is often impossible to implement in practice—achieved 31.22 dB. This indicates that the self-supervised framework recovers nearly as much information as models trained on explicit ground truth.

Qualitative Advantages

Beyond metrics, the model demonstrated superior continuity between slices. Error maps indicated significantly fewer artifacts and structural distortions compared to other state-of-the-art self-supervised methods like SMORE and TSCNet. The use of Implicit Neural Representations (INR) allows the model to treat the image as a continuous function, providing flexibility across various upsampling scales.

Conclusion and Future Work

This three-stage approach demonstrates that video-based pre-training and subject-specific fine-tuning can effectively overcome the data scarcity bottleneck in medical imaging. While the current implementation requires high-end hardware (e.g., NVIDIA A100) for the fastest fine-tuning, it offers a practical path toward generating high-fidelity 3D volumes from standard clinical 2D scans. Future research may extend this framework to other modalities, such as CT or PET scans, and explore ways to further bridge the domain gap between different MR weighting sequences.

Summary updated

Abstract

In clinical practice, 2D magnetic resonance (MR) sequences are widely adopted. While individual 2D slices can be stacked to form a 3D volume, the relatively large slice spacing can pose challenges for both image visualization and subsequent analysis tasks, which often require isotropic voxel spacing. To reduce slice spacing, deep-learning-based super-resolution techniques are widely investigated. However, most current solutions require a substantial number of paired high-resolution and low-resolution images for supervised training, which are typically unavailable in real-world scenarios. In this work, we propose a self-supervised super-resolution framework for inter-slice super-resolution of MR images. Our framework is first featured by pre-training on video dataset, as temporal correlation of videos is found beneficial for modeling the spatial relation among MR slices. Then, we use public high-quality MR dataset to fine-tune our pre-trained model, for enhancing awareness of our model to medical data. Finally, given a target dataset at hand, we utilize self-supervised fine-tuning to further ensure our model works well with user-specific super-resolution tasks. The proposed method demonstrates superior performance compared to other self-supervised methods and also holds the potential to benefit various downstream applications.
BibTeX
@inproceedings{wang2024inter,
  author = {Xin Wang and Zhiyun Song and Yitao Zhu and Sheng Wang and Lichi Zhang and Dinggang Shen and Qian Wang},
  title = {{Inter-slice Super-resolution of Magnetic Resonance Images by Pre-training and Self-supervised Fine-tuning}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=gPoUaOtB7s},
}
Figure