Learning to Rank for Blind Image Quality Assessment

Fei Gao, Dacheng Tao, Xinbo Gao, Xuelong Li

This paper, published in IEEE Transactions on Neural Networks and Learning Systems, presents a paradigm shift in how we teach computers to judge image quality. By moving away from noisy absolute scores and toward relative rankings, the authors developed a Blind Image Quality Assessment (BIQA) model that is more robust, easier to train, and highly scalable.

TL;DR

Instead of training models to predict arbitrary quality scores (e.g., "this image is a 74/100"), this paper uses "Learning to Rank" to teach models which of two images has better relative quality. By leveraging Preference Image Pairs (PIPs), the authors created a Blind Image Quality Assessment (BIQA) framework that matches state-of-the-art performance while being significantly easier to scale and generalize across different datasets.

What Problem Does This Paper Address?

Blind Image Quality Assessment (BIQA) aims to predict how a human would perceive the quality of an image without having a "perfect" reference image for comparison. Traditionally, this is treated as a regression problem: researchers collect thousands of images, ask humans to give them a score from 0 to 100 (Mean Opinion Score, or MOS), and train a model to predict that number.

However, MOS has several "human" flaws:

  1. Imprecision: Humans struggle to assign a precise number to quality; the difference between a 70 and a 75 is often arbitrary.
  2. Content Bias: People often give higher scores to "pretty" images (like a sunset) even if they have more technical distortion than a "boring" image (like a brick wall).
  3. Inconsistency: Scores vary wildly between different people and different experimental sessions.
  4. Scaling Nightmares: Combining two different image databases is difficult because their scoring scales rarely align.

Methods

The authors propose a Learning to Rank framework. Instead of asking "How good is this image?", the model asks "Is Image A better than Image B?"

1. Preference Image Pairs (PIPs)

The training data consists of pairs of images with a simple binary label: +1 if A is better than B, and -1 if B is better than A. These labels are much more consistent across different observers than absolute scores.

2. Feature Fusion

To represent the images, the authors fused three types of Natural Scene Statistics (NSS) features: BLIINDS-II, BRISQUE-L, and SRNSS. This 84-dimensional vector captures various aspects of image degradation, from DCT domain statistics to wavelet coefficients.

3. Multiple Kernel Learning (MKLGL)

To handle this fused feature set, they used a Multiple Kernel Learning algorithm based on Group Lasso (MKLGL). This allows the model to automatically weight different portions of the feature vector (representing different scales or properties) using 45 different Gaussian kernels.

4. Quality Prediction

To turn a ranking model back into a quality score, the authors introduced a "gain" strategy. A test image is compared against every image in the training set. Its final score is calculated based on how many "wins" it accumulates relative to the training images, mapped onto a standard 0-100 scale.

Datasets

The model was evaluated on the four largest standard IQA databases:

  • LIVE: 808 images with 5 distortion types.
  • TID2013: 3,000 images with 24 distortion types.
  • CSIQ: 866 images with 6 distortion types.
  • LIVEMD: 450 images with multiple simultaneous distortions.

Additionally, the authors conducted their own subjective study involving 240,000 image pair comparisons to validate that relative preferences are more reliable than absolute scores.

Key Insights and Ideas

  • Relative > Absolute: The "aha moment" is that humans are naturally better at comparing things than measuring them. A single observer's preference label is nearly as accurate as the consensus of a large group, which drastically lowers the cost of data collection.
  • Thresholding for Quality: When generating pairs from existing databases, the authors only used pairs where the score difference exceeded a threshold ($T$). This filters out "too close to call" pairs that would otherwise introduce noise into the training process.
  • Database Independence: Because the model learns relationships rather than scales, a model trained on one database (like LIVE) performs remarkably well on others (like CSIQ) without needing complex realignment.

Key Results

  • State-of-the-Art Performance: The proposed method achieved Spearman Rank Correlation Coefficient (SRCC) values comparable to or exceeding top-tier models like BRISQUE and CORNIA across all tested databases.
  • Robustness: The model showed high consistency across 100 random train-test trials, proving it isn't over-fitted to specific image contents.
  • Easy Extension: The authors demonstrated that adding a new distortion type to the model requires only a few hundred preference pairs, rather than a full-scale, expensive subjective study.

Limitations and Future Work

  • Computational Complexity: The "gain" prediction method requires comparing a test image against the entire training set. While effective, this could become a bottleneck if the training set grows to hundreds of thousands of images.
  • Feature Dependency: While the fused NSS features are powerful, they still struggle with certain complex distortions (like those in the LIVEMD database). The authors suggest that exploring more "quality-aware" features or deep learning-based features could be a future direction.
  • Efficiency: Future work could focus on more efficient ways to select which image pairs to label (active learning) to further reduce the human effort required for training.

Summary updated

Abstract

Blind image quality assessment (BIQA) aims to predict perceptual image quality scores without access to reference images. State-of-the-art BIQA methods typically require subjects to score a large number of images to train a robust model. However, subjective quality scores are imprecise, biased, and inconsistent, and it is challenging to obtain a large-scale database, or to extend existing databases, because of the inconvenience of collecting images, training the subjects, conducting subjective experiments, and realigning human quality evaluations. To combat these limitations, this paper explores and exploits preference image pairs (PIPs) such as the quality of image I <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">a</sub> is better than that of image I <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">b</sub> for training a robust BIQA model. The preference label, representing the relative quality of two images, is generally precise and consistent, and is not sensitive to image content, distortion type, or subject identity; such PIPs can be generated at a very low cost. The proposed BIQA method is one of learning to rank. We first formulate the problem of learning the mapping from the image features to the preference label as one of classification. In particular, we investigate the utilization of a multiple kernel learning algorithm based on group lasso to provide a solution. A simple but effective strategy to estimate perceptual image quality scores is then presented. Experiments show that the proposed BIQA method is highly effective and achieves a performance comparable with that of state-of-the-art BIQA algorithms. Moreover, the proposed method can be easily extended to new distortion categories.
BibTeX
@article{gao2015learning,
  author = {Fei Gao and Dacheng Tao and Xinbo Gao and Xuelong Li},
  title = {{Learning to Rank for Blind Image Quality Assessment}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2015},
  url = {https://openreview.net/forum?id=34g71BVh4ue},
}

Loading related content...

Figure