Abstract
Deep learning-based Image Quality Assessment (IQA) metrics are increasingly used to align computational quality scores with human perception. However, the lack of standardized pre-processing, particularly regarding image scaling, introduces significant performance variability. This study evaluates the stability of prominent Deep IQA models—LPIPS, PieAPP, and DISTS—across multiple scaling ratios and datasets. Results indicate that default scales are frequently suboptimal and that PieAPP exhibits the highest stability across varying resolutions, likely due to its patch-based training methodology.
Introduction
In computer vision, measuring image quality relative to a reference is essential. While traditional metrics like PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) are industry standards, they often fail to align with human visual perception. To bridge this gap, "Deep IQAs" such as LPIPS and DISTS utilize deep neural networks to mimic human judgment.
However, these models introduce a new challenge: sensitivity to spatial frequency and resolution. Currently, there is no standard for image resizing before IQA processing; some metrics use original resolutions while others down-scale. This inconsistency raises a stability problem: if a metric's score fluctuates significantly based solely on image size, its reliability as a benchmark is compromised.
Methodology
The authors conducted an empirical evaluation to determine how image scaling acts as an independent variable in IQA performance.
The Pipeline
The experimental setup involved taking pairs of reference and distorted images and applying ten different scaling ratios (from $1/9$ to $9/9$ of the original size) using bilinear interpolation. These scaled pairs were then processed by four popular Deep IQA models: LPIPS (with AlexNet and VGG backbones), PieAPP, and DISTS.
Datasets
To ensure generalizability, the study utilized five diverse datasets:
- TID2013 & KADID-10k: Focused on traditional distortions like noise and blur.
- PieAPP & PIPAL: Included complex distortions from GANs and super-resolution algorithms.
- CLIC2021: High-resolution images from a compression challenge.
Key Insights and Results
Suboptimality of Default Scales
A primary finding is that the "default" scaling suggested by metric authors—or simply using the original image size—is rarely the setting that yields the highest correlation with human judgment. For instance, the Spearman’s rank correlation (SRCC) for LPIPS (VGG-lin) on the TID2013 dataset improved by 0.172 when moving from the default scale to the optimal scale.
Scaling as a Perceptual Filter
Down-scaling often improved metric performance. The authors suggest that Deep IQAs can be overly sensitive to high-frequency distortions (e.g., multiplicative Gaussian noise). In these cases, down-scaling acts as a low-pass filter, allowing the model to focus on more perceptually relevant structures.
Stability Analysis
PieAPP emerged as the most stable model across all tested datasets. The authors hypothesize that this robustness stems from PieAPP's training on small $64 \times 64$ patches with random weight initialization, whereas other models rely on backbones pre-trained on ImageNet. This patch-based approach appears to make the model more invariant to global scale changes.
Conclusion and Future Work
The study demonstrates that image scaling is a critical factor in the performance of Deep IQA metrics. While PieAPP currently offers the most stable performance, the results highlight a need for native scale-invariance in future metric designs. Future research will explore the impact of different interpolation methods (e.g., bicubic or Lanczos) and conduct ablation studies to isolate the specific training factors that contribute to scale robustness.
Summary updated