A Perceptually Weighted Rank Correlation Indicator for Objective Image Quality Assessment

Qingbo Wu, Hongliang Li, Fanman Meng, King N. Ngan

Abstract

In the field of objective image quality assessment (IQA), Spearman's $\rho$ and Kendall's $\tau$ are the most popular rank correlation indicators. However, they assign uniform weight to all quality levels and assume each pair of images is sortable, ignoring two critical perceptual properties: the heightened importance of sorting accuracy (SA) for high-quality images and the subjective uncertainty in distinguishing perceptually similar images. This paper introduces a perceptually weighted rank correlation (PWRC) indicator that rewards the capability of correctly ranking high-quality images and suppresses attention toward insensitive rank mistakes. By focusing on "valid" pairwise comparisons that exceed a sensory threshold (ST), the proposed indicator offers a sophisticated SA-ST curve and an $AUC_{ca}$ metric for more robust IQA evaluation.

Introduction

Objective IQA metrics are essential for evaluating visual quality in computer vision. Traditionally, these metrics are validated by comparing their scores against human Mean Opinion Scores (MOS) using standard rank correlation coefficients. However, these standard statistics suffer from two major flaws. First, they treat all ranking errors equally, whereas real-world applications (like photo recommendation) prioritize the accuracy of top-ranked results. Second, they ignore the "perceptual threshold," penalizing algorithms for failing to distinguish between images that humans themselves find nearly identical.

Methodology

The PWRC indicator moves beyond a single "hard" correlation number toward a perceptually-aware evaluation framework built on three pillars:

  1. Nonuniform Weighting: An importance function $M(p, q)$ assigns higher weights to ranking mistakes involving high-quality images and large rank deviations.
  2. Selective Activation: A "soft" activation function $A(x, T)$ suppresses the impact of ranking errors when the quality difference between two images is below a given sensory threshold (ST).
  3. The SA-ST Curve and $AUC_{ca}$: Instead of a single coefficient, the authors plot Sorting Accuracy (SA) against a varying ST. The Area Under the Curve ($AUC_{ca}$) provides a "confidence-aware" metric of an algorithm's robustness across different levels of human visual sensitivity.

Key Insights: The Top-Rank Priority

A striking insight of this research is that two algorithms can have identical Spearman scores but vastly different practical utility. An algorithm that fails to identify the single best image is significantly worse for users than one that confuses the two worst images.

A high-quality parrot image (Rank 3). PWRC prioritizes the correct identification of such top-tier images.

Conversely, standard metrics often over-penalize errors in low-quality images, which are less critical for end-user satisfaction.

A low-quality, noisy parrot image (Rank 1). PWRC reduces the weight of ranking errors among such degraded samples.

By incorporating the standard deviation of human scores into the activation function, the authors mathematically model the "noise" of human judgment, preventing IQA models from being unfairly penalized for failing to solve perceptually "unsolvable" ranking tasks.

Results and Discussion

The researchers validated PWRC across five diverse IQA databases, including degraded images (LIVE II, TID2013) and enhanced images (IVCDehazing, ESPL-LIVE). Key findings include:

  • Superior Discriminative Power: PWRC successfully distinguished between ranking results that standard SRCC and KRCC scores found identical, correctly identifying which results were better for "pushing" high-quality images to users.
  • Consistency with $\Delta MOS$: The proposed $AUC_{ca}$ showed a much higher consistency with the average MOS difference than traditional metrics.
  • Robustness in Enhancement: On enhancement datasets, PWRC correctly identified robust IQA metrics (like BRISQUE) as superior selectors, whereas traditional Spearman correlation sometimes pointed toward less reliable metrics.

Conclusion

The PWRC indicator offers a new insight for interpreting visual perception behaviors and recommending robust IQA metrics. While the model currently relies on a constant derived from existing databases, it provides a more nuanced and practically useful framework for evaluating how well objective metrics align with human visual priorities. Future work may involve integrating PWRC directly into the training phase of neural networks to prioritize high-quality visual outputs.

Summary updated

Abstract

In the field of objective image quality assessment (IQA), the Spearman's $\rho$ and Kendall's $\tau$ are two most popular rank correlation indicators, which straightforwardly assign uniform weight to all quality levels and assume each pair of images are sortable. They are successful for measuring the average accuracy of an IQA metric in ranking multiple processed images. However, two important perceptual properties are ignored by them as well. Firstly, the sorting accuracy (SA) of high quality images are usually more important than the poor quality ones in many real world applications, where only the top-ranked images would be pushed to the users. Secondly, due to the subjective uncertainty in making judgement, two perceptually similar images are usually hardly sortable, whose ranks do not contribute to the evaluation of an IQA metric. To more accurately compare different IQA algorithms, we explore a perceptually weighted rank correlation indicator in this paper, which rewards the capability of correctly ranking high quality images, and suppresses the attention towards insensitive rank mistakes. More specifically, we focus on activating `valid' pairwise comparison towards image quality, whose difference exceeds a given sensory threshold (ST). Meanwhile, each image pair is assigned an unique weight, which is determined by both the quality level and rank deviation. By modifying the perception threshold, we can illustrate the sorting accuracy with a more sophisticated SA-ST curve, rather than a single rank correlation coefficient. The proposed indicator offers a new insight for interpreting visual perception behaviors. Furthermore, the applicability of our indicator is validated in recommending robust IQA metrics for both the degraded and enhanced image data.
BibTeX
@article{wu2018perceptually,
  author = {Qingbo Wu and Hongliang Li and Fanman Meng and King N. Ngan},
  title = {{A Perceptually Weighted Rank Correlation Indicator for Objective Image Quality Assessment}},
  journal = {IEEE Trans. Image Process.},
  year = {2018},
  url = {https://openreview.net/forum?id=3rCuC2c98l},
}
Figure