Relative Ranking for Perceptual Quality Assessment

35 concepts24 papers2 proposals

When we ask a human evaluator to rate the quality of an image on an absolute scale from 1 to 10, we introduce a massive amount of variance. An image of a beautiful sunset might receive an 8, while a technically flawless but mundane picture of a brick wall might get a 4. In this scenario, the evaluator is confounding content aesthetics with structural quality. Furthermore, human observers struggle with internal calibration; a 7 to one person might be a 5 to another, and these subjective scales shift depending on the environment and the sequence of images viewed.

Traditional perceptual quality assessment models attempt to mimic this flawed process by treating quality evaluation as an absolute regression task, mapping individual samples directly to a Mean Opinion Score (MOS). This approach frequently fails to generalize.

To solve this, a powerful paradigm shift has emerged: reframing quality assessment from absolute numerical scoring to relative pairwise ranking. By training models to answer the simpler question, "Which of these two samples is better?", researchers have unlocked models that align far more closely with human perception, generalize across diverse datasets, and can even learn without human-labeled data.

The Comparative Nature of Human Perception

The core intuition behind relative quality ranking is that human visual and auditory perception is inherently comparative. While assigning an absolute score to an isolated image is difficult, choosing which of two images looks more natural, sharp, or faithful to an original is an intuitive and highly consistent task.

In a foundational study on perceptual metrics, Richard Zhang leveraged this insight in The Unreasonable Effectiveness of Deep Features as a Perceptual Metric. By optimizing deep networks using a two-alternative forced choice (2AFC) objective—where the model must rank a reference-similar image higher than a distorted alternative—the author demonstrated that deep features naturally align with human visual similarity judgments when trained on comparative tasks.

Similarly, Ekta Prashnani et al. introduced PieAPP: Perceptual Image-Error Assessment Through Pairwise Preference, proving that training a model on pairwise preferences bypasses the noise and scale ambiguity of absolute numerical ratings. When human observers are asked to make relative judgments, their internal consistency skyrockets, providing a significantly cleaner training signal for machine learning models.

Absolute Regression (Unstable):
[Image A] ---> [ Neural Network ] ---> Score: 6.8 (Confounded by content)

Relative Ranking (Stable):
[Image A] ---\
              ---> [ Siamese / Rank Net ] ---> Preference: A > B (Isolates quality)
[Image B] ---/

Disentangling Content from Quality

Why does relative ranking perform so much better than absolute regression? The primary reason is that it disentangles scene content from degradation levels.

When evaluating complex tasks like image enhancement or retargeting, different algorithms produce distinct artifacts on the same source image. As Chen Yang et al. showed in Learning to Rank Retargeted Images, modeling the relative differences between pairs of retargeted results for the same source image allows the network to focus purely on the quality of the transformation, ignoring the underlying variations in image content.

This mechanism is particularly vital in highly distorted environments, such as underwater imaging, where severe color casts and murkiness skew absolute metrics. Chun-Le Guo et al. demonstrated in Underwater Ranker: Learn Which Is Better and How to Be Better that a margin ranking loss forces the network to learn the relative quality order of images within the same scene. This prevents the model from penalizing an enhanced image simply because the original scene was exceptionally dark or turbid. By isolating quality improvements from scene-specific variations, the ranker provides a robust training signal that can even be backpropagated to improve image enhancement networks themselves, a strategy also mirrored by Zhengyong Wang et al. in Domain Adaptation for Underwater Image Enhancement.

The benefits of this paradigm extend beyond vision. In speech processing, Alessandro Ragano et al. developed SCOREQ: Speech Quality Assessment with Contrastive Regression, replacing standard regression losses with a triplet ranking loss. They found that training models to recognize relative quality differences creates a more continuous, organized internal representation where the distance between feature embeddings directly maps to perceived degradation, leading to superior generalization across unseen audio domains.

Unifying Heterogeneous Scales and Self-Supervision

Reframing quality assessment as a ranking problem solves two major data bottlenecks: scale inconsistency across different datasets and the high cost of human annotation.

Resolving Scale Ambiguity

In blind image quality assessment (BIQA), models often need to learn from multiple independent datasets. However, combining these datasets is notoriously difficult because they use completely different scoring ranges and distributions. Weixia Zhang et al. solved this in Continual Learning for Blind Image Quality Assessment by employing a pairwise ranking loss. Because ranking only cares about whether image $A$ is better than image $B$, it completely bypasses the need for explicit score normalization, mapping heterogeneous subjective assessments into a single, unified perceptual scale.

Generating Infinite Training Pairs

Because ranking relies on relative differences, researchers can automatically synthesize training data without human labels. For example, Kede Ma et al. developed dipIQ: Blind Image Quality Assessment by Learning-to-Rank Discriminable Image Pairs, which uses objective full-reference metrics to automatically generate millions of "Quality-Discriminable Image Pairs" (DIPs). A blind neural network is then trained to predict which image has higher quality, achieving state-of-the-art robustness without a single human label.

Taking this further, Yue Huang et al. introduced Twice Mixing: A Rank Learning based Quality Assessment Approach for Underwater Image Enhancement. By mixing high-quality and low-quality versions of the same image at mathematically precise ratios, they created a self-supervised ranking hierarchy where the ground-truth quality order is known by construction, allowing the model to learn fine-grained perceptual degradation entirely on its own.

From Local Comparisons to Global Scores

If a model only learns to output a binary preference ($A > B$), how do we arrive at a continuous quality score for standard evaluation?

Mathematically, local pairwise comparisons can be aggregated into a consistent global quality hierarchy. As noted by Chen Yang et al., modeling relative differences between pairs is equivalent to global ranking based on subjective votes. In broader applications like urban planning, Frederico Freitas et al. demonstrated in Characterizing the perception of urban spaces from visual analytics of street-level imagery that competitive ranking algorithms can successfully aggregate crowdsourced pairwise preferences into highly stable global scores for individual images.

Pairwise Comparisons               Global Quality Scale
   A > B, B > C                    |-- Image A (Highest Quality)
   A > C, C > D   ============>    |-- Image B
   B > D                           |-- Image C
                                   |-- Image D (Lowest Quality)

Modern architectures often combine both worlds. The authors of No-Reference Image Quality Assessment via Transformers, Relative Ranking, and Self-Consistency showed that incorporating relative ranking objectives alongside absolute score regression enforces monotonicity and stability, preventing the model from making erratic predictions on slightly altered images.

Most recently, this paradigm has been adapted for Large Multimodal Models (LMMs). Baoliang Chen et al. developed Adaptive Image Quality Assessment via Teaching Large Multimodal Model to Compare, which prompts an LMM to rank target images against a set of predefined "anchor images." By leveraging the model's ability to compare rather than guess an absolute number, they successfully translate discrete linguistic preferences into highly precise, continuous quality scores.

By aligning the optimization objective with the comparative mechanics of human perception, relative quality ranking transforms a highly subjective, high-variance engineering challenge into a structured, robust, and scalable learning framework.

Key Foundational Work

Learning to Rank Retargeted Images (2017) — contributed 4 concept elements to this direction.
Quality Assessment for Comparing Image Enhancement Algorithms (2014) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_relative_quality_ranking_c29ea47a,
  title = {relative quality ranking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-quality-ranking/rdr_c29ea47a8c884fd6b6f44669ee01b019},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_relative_quality_ranking_c29ea47a_v1,
  title = {relative quality ranking},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/relative-quality-ranking/rdr_c29ea47a8c884fd6b6f44669ee01b019/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake