Evaluating the Stability of Deep Image Quality Assessment With Respect to Image Scaling

Koki Tsubota, Hiroaki Akutsu, Kiyoharu Aizawa

Abstract

Deep learning-based Image Quality Assessment (IQA) metrics are increasingly used to align computational quality scores with human perception. However, the lack of standardized pre-processing, particularly regarding image scaling, introduces significant performance variability. This study evaluates the stability of prominent Deep IQA models—LPIPS, PieAPP, and DISTS—across multiple scaling ratios and datasets. Results indicate that default scales are frequently suboptimal and that PieAPP exhibits the highest stability across varying resolutions, likely due to its patch-based training methodology.

Introduction

In computer vision, measuring image quality relative to a reference is essential. While traditional metrics like PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) are industry standards, they often fail to align with human visual perception. To bridge this gap, "Deep IQAs" such as LPIPS and DISTS utilize deep neural networks to mimic human judgment.

However, these models introduce a new challenge: sensitivity to spatial frequency and resolution. Currently, there is no standard for image resizing before IQA processing; some metrics use original resolutions while others down-scale. This inconsistency raises a stability problem: if a metric's score fluctuates significantly based solely on image size, its reliability as a benchmark is compromised.

Methodology

The authors conducted an empirical evaluation to determine how image scaling acts as an independent variable in IQA performance.

The Pipeline

The experimental setup involved taking pairs of reference and distorted images and applying ten different scaling ratios (from $1/9$ to $9/9$ of the original size) using bilinear interpolation. These scaled pairs were then processed by four popular Deep IQA models: LPIPS (with AlexNet and VGG backbones), PieAPP, and DISTS.

The experimental pipeline for evaluating IQA stability across different scaling ratios.

Datasets

To ensure generalizability, the study utilized five diverse datasets:

  • TID2013 & KADID-10k: Focused on traditional distortions like noise and blur.
  • PieAPP & PIPAL: Included complex distortions from GANs and super-resolution algorithms.
  • CLIC2021: High-resolution images from a compression challenge.

Key Insights and Results

Suboptimality of Default Scales

A primary finding is that the "default" scaling suggested by metric authors—or simply using the original image size—is rarely the setting that yields the highest correlation with human judgment. For instance, the Spearman’s rank correlation (SRCC) for LPIPS (VGG-lin) on the TID2013 dataset improved by 0.172 when moving from the default scale to the optimal scale.

Scaling as a Perceptual Filter

Down-scaling often improved metric performance. The authors suggest that Deep IQAs can be overly sensitive to high-frequency distortions (e.g., multiplicative Gaussian noise). In these cases, down-scaling acts as a low-pass filter, allowing the model to focus on more perceptually relevant structures.

Stability Analysis

PieAPP emerged as the most stable model across all tested datasets. The authors hypothesize that this robustness stems from PieAPP's training on small $64 \times 64$ patches with random weight initialization, whereas other models rely on backbones pre-trained on ImageNet. This patch-based approach appears to make the model more invariant to global scale changes.

Stability visualization across four Deep IQA models. PieAPP (bottom left) shows the tightest clustering near the origin, indicating the least performance variance across scales.

Conclusion and Future Work

The study demonstrates that image scaling is a critical factor in the performance of Deep IQA metrics. While PieAPP currently offers the most stable performance, the results highlight a need for native scale-invariance in future metric designs. Future research will explore the impact of different interpolation methods (e.g., bicubic or Lanczos) and conduct ablation studies to isolate the specific training factors that contribute to scale robustness.

Summary updated

Abstract

Image quality assessment (IQA) is a fundamental metric for image processing tasks (e.g., compression). With full-reference IQAs, traditional IQAs, such as PSNR and SSIM, have been used. Recently, IQAs based on deep neural networks (deep IQAs), such as LPIPS and DISTS, have also been used. It is known that image scaling is inconsistent among deep IQAs, as some perform down-scaling as pre-processing, whereas others instead use the original image size. In this paper, we show that the image scale is an influential factor that affects deep IQA performance. We comprehensively evaluate four deep IQAs on the same five datasets, and the experimental results show that image scale significantly influences IQA performance. We found that the most appropriate image scale is often neither the default nor the original size, and the choice differs depending on the methods and datasets used. We visualized the stability and found that PieAPP is the most stable among the four deep IQAs.
BibTeX
@article{tsubota2022evaluating,
  author = {Koki Tsubota and Hiroaki Akutsu and Kiyoharu Aizawa},
  title = {{Evaluating the Stability of Deep Image Quality Assessment With Respect to Image Scaling}},
  journal = {IEICE Trans. Information and Systems},
  year = {2022},
  doi = {10.1587/transinf.2022edl8025},
}
Figure