Abstract
Gathering human preferences for Image Quality Assessment (IQA) typically relies on forced-choice pairwise comparisons. While accurate, this method provides a localized training signal that can be noisy and lacks global context. This paper proposes a "rank-smoothed" loss function that integrates local pairwise data with a global ranking derived from the entire dataset. By utilizing Rank Centrality and a novel blending objective, the framework significantly improves model robustness and accuracy, achieving high correlation with human ratings on benchmark datasets.
Introduction
In Perceptual Quality Assessment, the gold standard for data collection is the forced-choice pairwise comparison, where human raters select the superior image from a pair. However, standard deep learning training methods, such as RankNet, often process these comparisons in small mini-batches. This "local" view fails to account for the global hierarchy of the dataset. For instance, a comparison between two low-quality images is treated with the same weight as a comparison between two high-quality images, ignoring their relative positions in the overall quality spectrum. Furthermore, human ratings are inherently noisy; a single outlier vote in a sparse dataset can disproportionately skew model learning. To address these issues, we introduce a framework that incorporates global context into the local learning process through rank-smoothing.
Methodology: The Rank-Smoothed Framework
The proposed framework bridges the gap between local empirical observations and global dataset structure through three primary components:
- Rank Centrality for Global Context: To establish a global quality metric, we employ the Rank Centrality algorithm. The dataset is modeled as a graph where images are nodes and pairwise comparisons are edges. By calculating the stationary distribution of a random walk on this graph—similar to the PageRank algorithm—each image is assigned a global quality score ($\omega$).
- $\beta$-Smoothing: To mitigate noise and prevent the distribution from becoming overly skewed toward a few top-tier images (especially in sparse data scenarios), a smoothing parameter ($\beta$) is applied to the global scores.
- Rank-Smoothed Loss Function: The core innovation is a composite objective function that blends two cross-entropy losses ($C_{ij}$):
- Local Loss ($C_{ij}^{local}$): Based on the direct empirical votes for a specific pair $(i, j)$.
- Global Loss ($C_{ij}^{global}$): Derived from the probabilities generated by the Rank Centrality scores.
A hyperparameter, $\alpha$, controls the trade-off between these signals. When $\alpha=1$, the model performs standard pairwise learning; as $\alpha$ decreases, the global ranking acts as an increasingly influential regularizer.
Experimental Evaluation
The model was evaluated using a combination of synthetic and large-scale real-world data:
- Google Photos Dataset: A massive dataset comprising 250,000 images and 17 million human responses to 3 million unique pairwise questions, focusing on technical attributes like sharpness and exposure.
- LIVE Dataset: A standard public benchmark used for cross-dataset evaluation to test the model's generalization capabilities.
Hyperparameter Sensitivity and Insights
Analysis of the hyperparameters $\alpha$ and $\beta$ reveals critical insights into the nature of pairwise learning. The research found that the optimal blending of signals typically occurs around $\alpha \approx 0.5$, suggesting that while global context is a powerful regularizer, local nuances remain essential for fine-grained quality discrimination.
As shown in the sensitivity plots, the global ranking becomes particularly beneficial as data becomes sparser (lower $r$). The global signal "smooths out" the noise inherent in individual human trials, correcting the training signal when a rater makes an accidental or inconsistent choice.
Results and Conclusion
The rank-smoothed loss consistently outperformed standard pairwise learning on large-scale subjective studies. Notably, the model demonstrated strong generalization, achieving a 0.71 linear correlation with human ratings on the LIVE dataset despite being trained on a different set of technical attributes. While the pre-computation of Rank Centrality scores adds a step to data preparation, the resulting model is appropriately sensitive to subtle technical flaws, matching human intuition. This approach is generic and holds potential for other fields involving pairwise learning, such as recommender systems and information retrieval.
Summary updated