An effective general-purpose NR-IQA model using natural scene statistics (NSS) of the luminance relative order

Tonghan Wang, Lu Zhang, Huizhen JiaSignal 2018 · DOI

TL;DR

This paper introduces a novel no-reference image quality assessment (NR-IQA) model that leverages natural scene statistics (NSS) derived from the "luminance relative order." By analyzing how distortions alter the statistical properties of image gradients, the model effectively predicts perceived image quality without needing a reference image, achieving state-of-the-art performance with reduced complexity.

What Problem Does This Paper Address?

Assessing image quality is crucial for various applications, from image compression and restoration to user experience in digital photography. Traditionally, image quality assessment (IQA) methods require a pristine "reference" image for comparison (Full-Reference, FR). However, in many real-world scenarios, such as streaming video or user-generated content, a reference image is unavailable. This necessitates the development of No-Reference (NR) IQA methods. While existing NR-IQA methods exist, they often fall into two categories: distortion-specific (DS), which require prior knowledge of the distortion type, or general-purpose, which aim to work across various distortions. The challenge lies in creating a general-purpose NR-IQA model that is both accurate and computationally efficient, capable of handling a wide range of distortions without prior information.

Methods

The proposed NR-IQA model is based on the observation that natural images exhibit predictable statistical properties, known as Natural Scene Statistics (NSS). Distortions, however, tend to disrupt these statistics. The core innovation here is the use of the "luminance relative order" of image gradients. Instead of directly analyzing gradients, the paper focuses on the relative order of luminance values between neighboring pixels.

The methodology involves several key steps:

  1. Contrast Normalization: The input image undergoes a local contrast normalization to decorrelate pixel values and approximate a unit normal Gaussian distribution, a common preprocessing step in NSS-based IQA.
  2. Feature Extraction: The model computes four types of features (variance, kurtosis, entropy, and differential entropy) from the log histograms of log directional derivatives (horizontal, vertical, main-diagonal, and secondary-diagonal). These features are extracted at two different scales (original and downsampled by half) to capture multiscale image characteristics. This results in a total of 32 features. The rationale is that the generalized Laplace model (GLM), which fits well with natural image statistics, is sensitive to variance and kurtosis, and entropy measures information content.
  3. Regression: A Support Vector Regression (SVR) model is trained to map the extracted 32-dimensional feature vector to a subjective quality score. SVR was chosen for its good prediction performance and relatively lower computational complexity compared to other methods like Random Forests.

Datasets

The paper evaluates its proposed method on several well-known IQA databases:

  • LIVE database II [28]: Contains 29 reference images with five types of distortions (JPEG 2000, JPEG, WN, Gblur, FF).
  • TID2008 database [41]: A larger database with 1700 test images derived from 25 reference images, featuring 17 types of distortions at four scales.
  • CSIQ IQA database [42]: Includes 30 reference images with six distortion types at various levels.
  • LIVE In the Wild Image Quality Challenge Database [48]: A large dataset of authentic image distortions from mobile devices.

The authors also perform database independence experiments, training on one database and testing on others (e.g., LIVE to CSIQ, LIVE to TID2008, etc.) to assess the model's generalization capabilities.

Key Insights and Ideas

  • Luminance Relative Order as a Powerful Feature: The paper highlights that the statistical properties of the "luminance relative order" of image gradients are highly sensitive to image distortions and effectively capture perceptual quality. This is a novel approach compared to directly using gradient magnitudes.
  • Generalized Laplace Model (GLM) Fit: The GLM proves to be a suitable model for characterizing the log histograms of log directional derivatives, providing a theoretical basis for using variance and kurtosis as features.
  • Multiscale Analysis is Crucial: Extracting features at multiple scales significantly improves the model's ability to capture image structure and distortion effects, aligning with how the human visual system perceives images.
  • SVR for Efficient Prediction: Support Vector Regression offers a good balance between prediction accuracy and computational efficiency, making it suitable for real-time applications.

Key Results

  • State-of-the-Art Performance: The proposed NR-IQA model achieves performance comparable to or better than existing state-of-the-art NR-IQA methods and even the full-reference SSIM metric on the LIVE database, as measured by SROCC and LCC.
  • Robustness Across Databases: The model demonstrates good generalization capabilities, performing well even when trained on one database and tested on another, indicating its robustness to variations in image content and distortion types.
  • Computational Efficiency: The proposed method is significantly faster than many other NR-IQA algorithms, being almost 173 times faster than FRIQUEE and only slightly slower than BRISQUE and GMLOG. This makes it a strong candidate for real-time applications.
  • Statistical Significance: T-tests confirm that the proposed method's SROCC values are statistically superior to most other tested algorithms.

Limitations and Future Work

  • Opinion-Awareness: The current model is "opinion-aware," meaning it requires training on human-rated distorted images. This limits its performance on new, unseen distortion types not present in the training data.
  • Future Work - Opinion-Unaware Models: The authors express a desire to extend their method to "opinion-unaware" (OU) or "distortion-unaware" (DU) NR-IQA metrics. They suggest that their relative order-based features might be suitable for such future developments, potentially leading to more generalizable and robust IQA models.
  • Biomedical Images: A related research [52] is mentioned, suggesting the potential application of NSS features for biomedical image quality assessment, which could be an avenue for future exploration.

Summary updated

BibTeX
@inproceedings{wang2018effective,
  author = {Tonghan Wang and Lu Zhang and Huizhen Jia},
  title = {{An effective general-purpose NR-IQA model using natural scene statistics (NSS) of the luminance relative order}},
  booktitle = {Signal},
  year = {2018},
  doi = {10.1016/j.image.2018.11.006},
}
Figure