Introduction
Deep learning has significantly advanced the field of Video Quality Assessment (VQA), which aims to predict human perception of video quality. However, two primary challenges persist: the scarcity of high-quality human-labeled data and the limitations of traditional regression-based training. Subjective human labels, often expressed as Mean Opinion Scores (MOS), are expensive and time-consuming to collect, resulting in small datasets that lead to overfitting. Furthermore, most models attempt to regress absolute quality scores, whereas the practical utility of VQA often lies in differentiating quality between two versions of the same content.
To address these issues, this paper introduces RankDVQA, a two-stage, ranking-inspired hybrid training framework. By leveraging a massive proxy-labeled dataset for patch-level ranking and a specialized spatio-temporal aggregation network, the model achieves state-of-the-art performance and superior generalization across diverse datasets without requiring per-database re-training.
The RankDVQA Framework
The RankDVQA architecture is divided into two distinct stages designed to handle different aspects of video quality perception: patch-level visual features and video-level temporal aggregation.
Stage 1: Patch-Level Quality (PQANet)
In the first stage, the Patch-wise Quality Assessment Network (PQANet) is trained to extract robust visual features. Instead of relying on scarce human labels, the authors utilize VMAF (Video Multi-Method Assessment Fusion) as a "teacher" to label a massive synthetic dataset of over 200,000 video patches. PQANet employs a Swin Transformer-based backbone and is trained using a Siamese ranking loss. Rather than predicting an absolute score, the network learns to identify which of two patches possesses higher quality, allowing it to learn a wide variety of distortion features from a vast dataset.
Stage 2: Spatio-Temporal Aggregation (STANet)
To transition from local patches to full-sequence quality, the Spatio-Temporal Aggregation Network (STANet) is introduced. STANet processes the patch-level indices and feature maps generated by PQANet. It utilizes 3D-CNNs to learn content-aware pooling, effectively weighting different spatial and temporal segments of the video based on their importance to perceived quality. This stage is trained on smaller, human-labeled datasets using a ranking-inspired loss to maintain the model's generalization capabilities.
Training Strategy and Datasets
The hybrid training strategy utilizes different data sources for each stage:
- Stage 1 Training: A large-scale database of 204,800 video groups derived from BVI-DVC and CLIC datasets, compressed with H.264, HEVC, AV1, and VVC codecs at various bitrates.
- Stage 2 Training: The VMAF+ and IVP databases, which contain subjective human scores.
- Evaluation: The model was evaluated on eight diverse HD benchmarks (including NFLX, BVI-HD, and MCL-V) without any fine-tuning, serving as a rigorous test of its zero-shot generalization.
Results and Discussion
RankDVQA demonstrates significant improvements over existing metrics in both Full-Reference (FR) and No-Reference (NR) configurations.
Key Findings:
- Superior Generalization: FR-RankDVQA achieved an average SROCC of 0.8972 across eight test sets, consistently outperforming the industry-standard VMAF (0.8644).
- High NR Performance: The No-Reference version (NR-RankDVQA) achieved an SROCC of 0.7791, outperforming several established Full-Reference metrics despite having no access to the original source video.
- Ranking over Regression: By training on VMAF rankings rather than absolute scores, the model avoids inheriting specific biases of the proxy metric while benefiting from its reliability in comparative scenarios.
Conclusion
RankDVQA effectively overcomes the data bottleneck in deep VQA through a ranking-inspired hybrid training approach. By separating feature learning from temporal aggregation and utilizing proxy-labeled ranking data, the framework achieves high accuracy and remarkable cross-dataset stability. While the model currently focuses on HD content and carries a higher computational cost than traditional metrics like VMAF, it sets a new benchmark for robust, deep-learning-based video quality assessment. Future work will focus on optimizing the transformer backbone for real-time applications and extending the framework to 4K and 8K resolutions.
Summary updated