Rank-Smoothed Pairwise Learning In Perceptual Quality Assessment

Hossein Talebi, Ehsan Amid, Peyman Milanfar, Manfred K. Warmuth

Abstract

Gathering human preferences for Image Quality Assessment (IQA) typically relies on forced-choice pairwise comparisons. While accurate, this method provides a localized training signal that can be noisy and lacks global context. This paper proposes a "rank-smoothed" loss function that integrates local pairwise data with a global ranking derived from the entire dataset. By utilizing Rank Centrality and a novel blending objective, the framework significantly improves model robustness and accuracy, achieving high correlation with human ratings on benchmark datasets.

Introduction

In Perceptual Quality Assessment, the gold standard for data collection is the forced-choice pairwise comparison, where human raters select the superior image from a pair. However, standard deep learning training methods, such as RankNet, often process these comparisons in small mini-batches. This "local" view fails to account for the global hierarchy of the dataset. For instance, a comparison between two low-quality images is treated with the same weight as a comparison between two high-quality images, ignoring their relative positions in the overall quality spectrum. Furthermore, human ratings are inherently noisy; a single outlier vote in a sparse dataset can disproportionately skew model learning. To address these issues, we introduce a framework that incorporates global context into the local learning process through rank-smoothing.

Methodology: The Rank-Smoothed Framework

The proposed framework bridges the gap between local empirical observations and global dataset structure through three primary components:

  1. Rank Centrality for Global Context: To establish a global quality metric, we employ the Rank Centrality algorithm. The dataset is modeled as a graph where images are nodes and pairwise comparisons are edges. By calculating the stationary distribution of a random walk on this graph—similar to the PageRank algorithm—each image is assigned a global quality score ($\omega$).
  2. $\beta$-Smoothing: To mitigate noise and prevent the distribution from becoming overly skewed toward a few top-tier images (especially in sparse data scenarios), a smoothing parameter ($\beta$) is applied to the global scores.
  3. Rank-Smoothed Loss Function: The core innovation is a composite objective function that blends two cross-entropy losses ($C_{ij}$):
  4. Local Loss ($C_{ij}^{local}$): Based on the direct empirical votes for a specific pair $(i, j)$.
  5. Global Loss ($C_{ij}^{global}$): Derived from the probabilities generated by the Rank Centrality scores.
The rank-smoothed learning pipeline, showing the integration of local pairwise comparisons and global rank aggregation into a unified loss function for CNN training.

A hyperparameter, $\alpha$, controls the trade-off between these signals. When $\alpha=1$, the model performs standard pairwise learning; as $\alpha$ decreases, the global ranking acts as an increasingly influential regularizer.

Experimental Evaluation

The model was evaluated using a combination of synthetic and large-scale real-world data:

  • Google Photos Dataset: A massive dataset comprising 250,000 images and 17 million human responses to 3 million unique pairwise questions, focusing on technical attributes like sharpness and exposure.
  • LIVE Dataset: A standard public benchmark used for cross-dataset evaluation to test the model's generalization capabilities.

Hyperparameter Sensitivity and Insights

Analysis of the hyperparameters $\alpha$ and $\beta$ reveals critical insights into the nature of pairwise learning. The research found that the optimal blending of signals typically occurs around $\alpha \approx 0.5$, suggesting that while global context is a powerful regularizer, local nuances remain essential for fine-grained quality discrimination.

Sensitivity analysis showing the effect of the number of comparisons ($n_t$) and data sparsity ($r$) on the optimal values for the blending parameter $\alpha$ and the smoothing parameter $\beta$.

As shown in the sensitivity plots, the global ranking becomes particularly beneficial as data becomes sparser (lower $r$). The global signal "smooths out" the noise inherent in individual human trials, correcting the training signal when a rater makes an accidental or inconsistent choice.

Results and Conclusion

The rank-smoothed loss consistently outperformed standard pairwise learning on large-scale subjective studies. Notably, the model demonstrated strong generalization, achieving a 0.71 linear correlation with human ratings on the LIVE dataset despite being trained on a different set of technical attributes. While the pre-computation of Rank Centrality scores adds a step to data preparation, the resulting model is appropriately sensitive to subtle technical flaws, matching human intuition. This approach is generic and holds potential for other fields involving pairwise learning, such as recommender systems and information retrieval.

Summary updated

Abstract

Conducting pairwise comparisons is a widely used approach in curating human perceptual preference data. Typically raters are instructed to make their choices according to a specific set of rules that address certain dimensions of image quality and aesthetics. The outcome of this process is a dataset of sampled image pairs with their associated empirical preference probabilities. Training a model on these pairwise preferences is a common deep learning approach. However, optimizing by gradient descent through mini-batch learning means that the “global” ranking of the images is not explicitly taken into account. In other words, each step of the gradient descent relies only on a limited number of pairwise comparisons. In this work, we demonstrate that regularizing the pairwise empirical probabilities with aggregated rankwise probabilities leads to a more reliable training loss. We show that training a deep image quality assessment model with our rank-smoothed loss consistently improves the accuracy of predicting human preferences.
BibTeX
@inproceedings{talebi2020rank,
  author = {Hossein Talebi and Ehsan Amid and Peyman Milanfar and Manfred K. Warmuth},
  title = {{Rank-Smoothed Pairwise Learning In Perceptual Quality Assessment}},
  booktitle = {ICIP},
  year = {2020},
  url = {https://openreview.net/forum?id=MuWoiTQJLj3},
}
Figure