Inter-Slice Super-Resolution of Magnetic Resonance Images by Pre-Training and Self-Supervised Fine-Tuning

Xin Wang, Zhiyun Song, Yitao Zhu, Sheng Wang, Lichi Zhang, Dinggang Shen, Qian Wang

TL;DR

Researchers have developed a three-stage framework to boost the resolution of MRI scans without needing expensive, high-resolution ground truth data for every patient. By pre-training on natural videos to learn spatial flow and then fine-tuning on public medical data and specific patient scans, the model achieves near-supervised performance in turning "thick-slice" 2D scans into high-fidelity 3D volumes.

What Problem Does This Paper Address?

In clinical settings, doctors often use 2D Magnetic Resonance (MR) sequences because they are fast to acquire. However, these scans usually have a "thick-slice" problem: while the resolution within a slice is high, the gap between slices is large. This creates "anisotropic" voxels (think of them as tall, skinny rectangular prisms instead of perfect cubes), which makes it difficult for doctors to visualize anatomy from different angles or for AI software to perform automated analysis.

While deep learning can "fill in the gaps" (Super-Resolution or SR), most models require thousands of paired low-resolution (LR) and high-resolution (HR) images to learn. In the real world, we rarely have the HR version of a clinical scan. Existing self-supervised methods try to get around this, but they often struggle with limited data or produce blurry results.

Methods

The authors propose a clever three-stage pipeline to bridge the gap between "no data" and "high performance":

  1. Stage 1: Pre-training on Video (Video Temporal SR): The researchers realized that a sequence of MR slices is mathematically similar to a sequence of video frames. They pre-trained a Spatially-Aware Implicit Neural Representation (SA-INR) model on a large dataset of natural videos. This teaches the model how to interpolate "missing" frames by understanding temporal and spatial correlations.
  2. Stage 2: Fine-tuning on High-quality MR: To move from "cats and cars" to "bones and tissues," they fine-tuned the model on a public dataset of high-quality, thin-slice MRIs. This helps the model learn the specific textures and anatomical structures of medical imaging.
  3. Stage 3: Self-supervised Fine-tuning (SSF) on Target Data: Finally, when given a specific patient's low-resolution scan, the model performs a "subject-specific" tune-up. It downsamples the existing LR scan even further to create a "proxy" training set, allowing the model to adapt to the specific noise and contrast of that individual scan without needing an external ground truth.

Datasets

  • REDS-VTSR: A large-scale video dataset used for the initial "temporal" pre-training.
  • Osteoarthritis Initiative (OAI): A public dataset of 350 high-quality 3D MR volumes used for the medical domain adaptation stage.
  • SKI10 (Segmentation of Knee Images 2010): The target dataset used for evaluation, consisting of 150 clinical-grade knee scans.

Key Insights and Ideas

  • Videos as a Proxy for Anatomy: The most striking insight is that learning how objects move in a video helps a model understand how anatomical structures "move" or change across 3D slices. This "temporal-to-spatial" transfer learning significantly reduces the amount of medical data needed.
  • The Power of Subject-Specific Tuning: Even a model trained on thousands of other people's knees might struggle with a new patient due to different scanner settings or anatomy. By spending just ~1 minute fine-tuning on the specific patient's data (Stage 3), the model significantly reduces errors.
  • Implicit Neural Representations (INR): Unlike traditional grids, the SA-INR model treats the image as a continuous function. This allows it to predict values at any coordinate, making it much more flexible for different upsampling scales.

Key Results

The proposed method outperformed several state-of-the-art self-supervised methods (like SMORE and TSCNet) across both quantitative and qualitative measures:

  • Quantitative: On the SKI10 dataset, the model achieved a PSNR (Peak Signal-to-Noise Ratio) of 30.88 dB. For context, a fully supervised model (which is the "gold standard" but impossible in many clinical cases) achieved 31.22 dB. The proposed method is remarkably close to the upper bound.
  • Qualitative: Error maps showed that the proposed method produced much fewer artifacts and "distorted structures" compared to previous methods (see Fig. 2 in the paper), maintaining better continuity between slices.
  • Efficiency: The subject-based fine-tuning is fast enough for practical use, taking about 1 minute on a modern GPU (NVIDIA A100), with the actual super-resolution process taking only 20 seconds.

Limitations and Future Work

While the results are impressive, the paper notes a few areas for growth:

  • Domain Gap: While Stage 2 uses public MR data, there is still a "domain gap" between different types of MR sequences (e.g., T1-weighted vs. T2-weighted).
  • Computational Requirements: Although 1 minute is fast, it still requires a high-end GPU (A100) for the fine-tuning stage, which might not be available in every clinical workstation.
  • Future Directions: The authors suggest this framework could be extended to other 3D medical imaging tasks beyond just the knee, potentially benefiting CT or PET scans where slice thickness is also a concern.

Summary updated

Abstract

In clinical practice, 2D magnetic resonance (MR) sequences are widely adopted. While individual 2D slices can be stacked to form a 3D volume, the relatively large slice spacing can pose challenges for both image visualization and subsequent analysis tasks, which often require isotropic voxel spacing. To reduce slice spacing, deep-learning-based super-resolution techniques are widely investigated. However, most current solutions require a substantial number of paired high-resolution and low-resolution images for supervised training, which are typically unavailable in real-world scenarios. In this work, we propose a self-supervised super-resolution framework for inter-slice super-resolution of MR images. Our framework is first featured by pre-training on video dataset, as temporal correlation of videos is found beneficial for modeling the spatial relation among MR slices. Then, we use public high-quality MR dataset to fine-tune our pre-trained model, for enhancing awareness of our model to medical data. Finally, given a target dataset at hand, we utilize self-supervised fine-tuning to further ensure our model works well with user-specific super-resolution tasks. The proposed method demonstrates superior performance compared to other self-supervised methods and also holds the potential to benefit various downstream applications.
BibTeX
@inproceedings{wang2024inter,
  author = {Xin Wang and Zhiyun Song and Yitao Zhu and Sheng Wang and Lichi Zhang and Dinggang Shen and Qian Wang},
  title = {{Inter-Slice Super-Resolution of Magnetic Resonance Images by Pre-Training and Self-Supervised Fine-Tuning}},
  booktitle = {ISBI},
  year = {2024},
  url = {https://openreview.net/forum?id=gPoUaOtB7s},
}

Loading related content...

Figure