Image Quality Assessment for Magnetic Resonance Imaging

Sergey Kastryulin, Jamil Zakirov, Nicola Pezzotti, Dmitry V. Dylov

TL;DR

This paper tackles the crucial problem of accurately assessing the quality of Magnetic Resonance Imaging (MRI) scans, a task where standard image quality metrics often fall short. By conducting an extensive evaluation with expert radiologists and correlating their judgments with 35 different image quality metrics, the researchers identify the most reliable metrics for MRI applications, particularly for reconstructed images from accelerated scans, motion correction, and denoising.

What Problem Does This Paper Address?

Image Quality Assessment (IQA) is vital for evaluating the performance of image processing algorithms. However, most existing IQA metrics are designed for general image domains and often tested with simple, artificial distortions like added noise or blurring. This approach doesn't accurately reflect the complex nuances of medical imaging, especially MRI, where specific diagnostic features are paramount. Radiologists' perception of MRI quality is influenced by factors like signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR), and the presence of artifacts, which are often coupled and not well-captured by generic metrics. Furthermore, evaluating these metrics on outputs from real-world deep learning models used in MRI (like those for scan acceleration or denoising) is crucial, as these models introduce imperfections that differ from simple artificial distortions. This paper aims to bridge this gap by rigorously evaluating IQA metrics specifically for MRI, focusing on radiologist perception and real-world deep learning model outputs.

Methods

The core of this research lies in a comprehensive evaluation of 35 image quality metrics (including full-reference, no-reference, and distribution-based types). The authors generated corrupted MRI data by applying scan acceleration, motion, and denoising using deep learning models trained on the FastMRI dataset. Instead of using simple artificial distortions, they used the outputs of these models, which represent more realistic imperfections. Seven experienced radiologists then assessed these reconstructed images based on three key criteria: perceived noise (SNR), perceived contrast (CNR), and presence of artifacts. Their subjective scores were then correlated with the objective scores from the 35 IQA metrics. To handle the distribution-based metrics, which typically compare sets of images, the researchers adapted them by cropping images into overlapping tiles. The correlation analysis used Spearman's rank-order correlation coefficient (SRCC) and Kendall's rank-order correlation coefficient (KRCC), along with non-linear regression to better fit the subjective scores.

Datasets

The study utilized the FastMRI dataset, a large repository of raw multi-coil MRI k-space data for both knee and brain scans. This dataset was used to generate corrupted images by applying scan acceleration, motion, and denoising using trained neural network models. The generated corrupted images, along with their corresponding clean reference images, formed the basis for the labeling dataset.

Key Insights and Ideas

  • Domain Shift Matters: Standard IQA metrics, developed for natural images, often perform poorly when applied to medical images like MRI. The ranks of metrics can change significantly between natural and MRI domains.
  • Radiologist Perception is Key: Simply measuring pixel-wise differences isn't enough. Metrics that correlate well with radiologists' judgments on specific criteria (SNR, CNR, artifacts) are essential for MRI.
  • Real-world Distortions are Crucial: Evaluating IQA metrics on outputs from deep learning models used in MRI (e.g., for acceleration or denoising) provides a more realistic assessment than using simple, artificially introduced distortions.
  • Feature Extractors Impact DB Metrics: For distribution-based metrics, the choice of feature extractor (e.g., Inception vs. VGG) significantly influences their performance. VGG-based extractors often show better correlation.
  • No Single "Best" Metric: While some metrics perform better overall, their effectiveness can vary depending on the specific MRI task (acceleration, denoising, motion correction) and anatomy (brain, knee).

Key Results

  • The study evaluated 14,700 subjective scores from 7 trained radiologists.
  • Top-performing metrics that showed good correlation with radiologist perception across the evaluated criteria were DISTS, HaarPSI, VSI, and FIDVGG16.
  • These top metrics were found to be efficient across the three quality criteria (SNR, CNR, artifacts), for both knee and brain anatomies, and for the target tasks of scan acceleration, motion correction, and denoising.
  • Conventional metrics like PSNR and SSIM were found to be less reliable, often misjudging image quality due to simple brightness shifts or blurring, while more advanced metrics better reflected visual perception.
  • The study found that feature extractors like VGG16 often outperformed InceptionV3 for distribution-based metrics in the MRI context.

Limitations and Future Work

  • Labeling Discrepancies: While efforts were made to ensure radiologist self-consistency (e.g., through re-labeling sessions), some level of discrepancy in subjective scoring is inherent. The study found no significant correlation between labeling time and self-consistency, suggesting other factors like individual experience play a larger role.
  • Need for MRI-Specific Metrics: The paper concludes that an optimal MRI-specific metric is yet to be devised. Future work should focus on developing more interpretable metrics that can model MRI-specific perception and incorporate formalized rules from medical textbooks.
  • Expanding NR Metrics: While current no-reference (NR) metrics showed mediocre performance in this study, the authors believe their value will increase with more available data, drawing parallels to the progress seen in natural image domains.
  • Task-Specific IQA: The study highlights the need for metrics tailored to specific downstream tasks in MRI, rather than relying on general quality assessment.

Summary updated

Abstract

Image quality assessment (IQA) algorithms aim to reproduce the human’s perception of the image quality. The growing popularity of image enhancement, generation, and recovery models instigated the development of many methods to assess their performance. However, most IQA solutions are designed to predict image quality in the general domain, with the applicability to specific areas, such as medical imaging, remaining questionable. Moreover, the selection of these IQA metrics for a specific task typically involves intentionally induced distortions, such as manually added noise or artificial blurring; yet, the chosen metrics are then used to judge the output of real-life computer vision models. In this work, we aspire to fill these gaps by carrying out the most extensive IQA evaluation study for Magnetic Resonance Imaging (MRI) to date (14,700 subjective scores). We use outputs of neural network models trained to solve problems relevant to MRI, including image reconstruction in the scan acceleration, motion correction, and denoising. Our emphasis is on reflecting the radiologist’s perception of the reconstructed images, gauging the most diagnostically influential criteria for the quality of MRI scans: signal-to-noise ratio, contrast-to-noise ratio, and the presence of art efacts. Seven trained radiologists assess these distorted images, with their verdicts then correlated with 35 different image quality metrics (full-reference, no-reference, and distribution-based metrics considered). The top performers– DISTS, HaarPSI, VSI, and FIDVGG16– are found to be efficient across three proposed quality criteria, for all considered anatomies and the target tasks.
BibTeX
@inproceedings{kastryulin2023image,
  author = {Sergey Kastryulin and Jamil Zakirov and Nicola Pezzotti and Dmitry V. Dylov},
  title = {{Image Quality Assessment for Magnetic Resonance Imaging}},
  booktitle = {IEEE Access},
  year = {2023},
  url = {https://openreview.net/forum?id=3Ps3hMmPUDZ},
}

Loading related content...

Figure