RankDVQA: Deep VQA based on Ranking-inspired Hybrid Training

Chen Feng, Duolikun Danier, Fan Zhang, David Bull

Introduction

Deep learning has significantly advanced the field of Video Quality Assessment (VQA), which aims to predict human perception of video quality. However, two primary challenges persist: the scarcity of high-quality human-labeled data and the limitations of traditional regression-based training. Subjective human labels, often expressed as Mean Opinion Scores (MOS), are expensive and time-consuming to collect, resulting in small datasets that lead to overfitting. Furthermore, most models attempt to regress absolute quality scores, whereas the practical utility of VQA often lies in differentiating quality between two versions of the same content.

To address these issues, this paper introduces RankDVQA, a two-stage, ranking-inspired hybrid training framework. By leveraging a massive proxy-labeled dataset for patch-level ranking and a specialized spatio-temporal aggregation network, the model achieves state-of-the-art performance and superior generalization across diverse datasets without requiring per-database re-training.

The RankDVQA Framework

The RankDVQA architecture is divided into two distinct stages designed to handle different aspects of video quality perception: patch-level visual features and video-level temporal aggregation.

The RankDVQA architecture, illustrating the two-stage framework: PQANet for patch-level quality and STANet for spatio-temporal aggregation.

Stage 1: Patch-Level Quality (PQANet)

In the first stage, the Patch-wise Quality Assessment Network (PQANet) is trained to extract robust visual features. Instead of relying on scarce human labels, the authors utilize VMAF (Video Multi-Method Assessment Fusion) as a "teacher" to label a massive synthetic dataset of over 200,000 video patches. PQANet employs a Swin Transformer-based backbone and is trained using a Siamese ranking loss. Rather than predicting an absolute score, the network learns to identify which of two patches possesses higher quality, allowing it to learn a wide variety of distortion features from a vast dataset.

Stage 2: Spatio-Temporal Aggregation (STANet)

To transition from local patches to full-sequence quality, the Spatio-Temporal Aggregation Network (STANet) is introduced. STANet processes the patch-level indices and feature maps generated by PQANet. It utilizes 3D-CNNs to learn content-aware pooling, effectively weighting different spatial and temporal segments of the video based on their importance to perceived quality. This stage is trained on smaller, human-labeled datasets using a ranking-inspired loss to maintain the model's generalization capabilities.

Training Strategy and Datasets

The hybrid training strategy utilizes different data sources for each stage:

  • Stage 1 Training: A large-scale database of 204,800 video groups derived from BVI-DVC and CLIC datasets, compressed with H.264, HEVC, AV1, and VVC codecs at various bitrates.
  • Stage 2 Training: The VMAF+ and IVP databases, which contain subjective human scores.
  • Evaluation: The model was evaluated on eight diverse HD benchmarks (including NFLX, BVI-HD, and MCL-V) without any fine-tuning, serving as a rigorous test of its zero-shot generalization.

Results and Discussion

RankDVQA demonstrates significant improvements over existing metrics in both Full-Reference (FR) and No-Reference (NR) configurations.

Performance comparison of RankDVQA against state-of-the-art VQA metrics using Spearman Rank Order Correlation Coefficient (SROCC).

Key Findings:

  • Superior Generalization: FR-RankDVQA achieved an average SROCC of 0.8972 across eight test sets, consistently outperforming the industry-standard VMAF (0.8644).
  • High NR Performance: The No-Reference version (NR-RankDVQA) achieved an SROCC of 0.7791, outperforming several established Full-Reference metrics despite having no access to the original source video.
  • Ranking over Regression: By training on VMAF rankings rather than absolute scores, the model avoids inheriting specific biases of the proxy metric while benefiting from its reliability in comparative scenarios.

Conclusion

RankDVQA effectively overcomes the data bottleneck in deep VQA through a ranking-inspired hybrid training approach. By separating feature learning from temporal aggregation and utilizing proxy-labeled ranking data, the framework achieves high accuracy and remarkable cross-dataset stability. While the model currently focuses on HD content and carries a higher computational cost than traditional metrics like VMAF, it sets a new benchmark for robust, deep-learning-based video quality assessment. Future work will focus on optimizing the transformer backbone for real-time applications and extending the framework to 4K and 8K resolutions.

Summary updated

Abstract

In recent years, deep learning techniques have shown significant potential for improving video quality assessment (VQA), achieving higher correlation with subjective opinions compared to conventional approaches. However, the development of deep VQA methods has been constrained by the limited availability of large-scale training databases and ineffective training methodologies. As a result, it is difficult for deep VQA approaches to achieve consistently superior performance and model generalization. In this context, this paper proposes new VQA methods based on a two-stage training methodology which motivates us to develop a large-scale VQA training database without employing human subjects to provide ground truth labels. This method was used to train a new transformer-based network architecture, exploiting quality ranking of different distorted sequences rather than minimizing the difference from the ground-truth quality labels. The resulting deep VQA methods (for both full reference and no reference scenarios), FR-and NR-RankDVQA, exhibit consistently higher correlation with perceptual quality compared to the state-of-the-art conventional and deep VQA methods, with average SROCC values of 0.8972 (FR) and 0.7791 (NR) over eight test sets without performing cross-validation.
BibTeX
@inproceedings{feng2024rankdvqa,
  author = {Chen Feng and Duolikun Danier and Fan Zhang and David Bull},
  title = {{RankDVQA: Deep VQA based on Ranking-inspired Hybrid Training}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00167},
}
Figure