Regression or classification? New methods to evaluate no-reference picture and video quality models

Zhengzhong Tu, Chia-Ju Chen, Li-Heng Chen, Yilin Wang, Neil Birkbeck, Balu Adsumilli, Alan C. Bovik

Introduction

No-Reference Quality Assessment (NR-QA) has traditionally been treated as a regression problem: predicting a continuous Mean Opinion Score (MOS) for an image or video. However, for User-Generated Content (UGC), which often contains complex, overlapping distortions like blur, noise, and compression artifacts, predicting an exact score is notoriously difficult.

This paper challenges the status quo, arguing that fine-grained regression may be overkill for many practical applications. For instance, a streaming platform primarily needs to know if a video is "High" or "Low" quality to decide on an encoding strategy. By reframing quality prediction as binary or ordinal classification, the authors propose a more tractable approach that aligns better with industry needs.

Methodology: From Scores to Classes

The researchers propose two alternative tasks to the standard regression approach:

  1. Binary Classification: Categorizing content into "High" vs. "Low" quality.
  2. Ordinal Classification: Categorizing content into a three-point scale: {Low, Medium, High}.

To bridge the gap between continuous MOS and discrete labels without using arbitrary midpoints, the authors employ a Gaussian Mixture Model (GMM). The GMM fits the empirical distribution of MOS scores to identify natural clusters of quality, allowing for data-driven, adaptive thresholds.

Figure 1: Visualization of the GMM-based thresholding approach. The model fits the MOS histogram to define "Low" and "High" quality clusters.

The study evaluates various models, including "hand-crafted" feature-based methods (e.g., BRISQUE, FRIQUEE) and deep learning baselines (VGG-19, ResNet-50), using Support Vector Machines (SVM) as the backend for both classification and regression.

Benchmarking across UGC Datasets

The evaluation spans six major UGC datasets representing "authentic" distortions, moving away from synthetic laboratory settings:

  • Images (IQA): CLIVE, KonIQ-10k, and SPAQ.
  • Videos (VQA): KoNViD-1k, LIVE-VQC, and YouTube-UGC.

Key Results and Insights

The benchmarking effort revealed that the "best" model often changes depending on the task. A model excelling at ranking (high SRCC in regression) might not be the top performer in binary classification. This suggests that different evaluation tasks reveal distinct strengths and weaknesses in quality models.

Figure 3: Performance comparison of various VQA models across regression, binary classification, and ordinal classification tasks on three major video datasets.

Key findings include:

  • Task-Dependent Performance: On the CLIVE image dataset, ResNet-50 led in regression, but FRIQUEE was superior for binary classification, and HOSA took the lead in ordinal classification.
  • Video Consistency: Results were more stable on video datasets, with TLVQM, ResNet-50, and VIDEVAL leading on LIVE-VQC, KoNViD-1k, and YouTube-UGC respectively across nearly all metrics.
  • High Accuracy: Binary classification accuracies reached 75% to 89%, suggesting that even simpler models can reliably distinguish between high and low quality.

Conclusion and Future Directions

Relaxing NR-QA into a classification task makes models more robust to the inherent noise in human subjective ratings and maps directly onto industry frameworks like Quality-Guided Transcoding (QGT). While current results are promising, future work should explore end-to-end training with ordinal loss functions and test cross-dataset generalization to further validate these methods for real-world deployment.

Summary updated

BibTeX
@inproceedings{tu2021regression,
  author = {Zhengzhong Tu and Chia-Ju Chen and Li-Heng Chen and Yilin Wang and Neil Birkbeck and Balu Adsumilli and Alan C. Bovik},
  title = {{Regression or classification? New methods to evaluate no-reference picture and video quality models}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=ffRt51orKhx},
}
Figure