TL;DR
This paper introduces RankDVQA, a novel deep learning approach for video quality assessment (VQA) that leverages a ranking-inspired training methodology. By creating a large-scale, automatically annotated dataset and training models to predict relative quality rather than absolute scores, RankDVQA achieves state-of-the-art performance and superior generalization across various VQA benchmarks.
What Problem Does This Paper Address?
Video quality assessment (VQA) is crucial for ensuring a good user experience in the age of streaming. While deep learning has shown promise in VQA, its progress has been hampered by two main issues: a lack of large, diverse training datasets and ineffective training methods. Existing deep VQA models often struggle to generalize well to new datasets because they are trained on limited data and focus on predicting absolute quality scores, which can be sensitive to variations in experimental setups. This paper aims to overcome these limitations by proposing a new training paradigm and a novel dataset that enable more robust and generalizable VQA models.
Methods
The core innovation of RankDVQA lies in its ranking-inspired hybrid training methodology, which consists of two stages:
- Stage 1: Patch-Level Quality Assessment (PQANet): This stage employs a transformer-based network (specifically, a Swin Transformer architecture) to predict quality indices for individual video patches. Instead of minimizing the difference to ground-truth labels, PQANet is trained to predict the relative quality between pairs of patches. This is achieved using a Siamese network structure and a binary cross-entropy loss based on VMAF-generated ranking labels.
- Stage 2: Spatio-Temporal Aggregation Network (STANet): This stage takes the patch-level quality indices from PQANet and combines them with extracted spatio-temporal features to produce a final, sequence-level quality score. STANet is trained using a ranking-inspired loss that considers the difference in subjective scores between pairs of sequences, allowing for the combination of multiple VQA databases.
A key aspect of this approach is the creation of a large-scale training dataset (204,800 video dodecuplet groups) without relying on costly human subjective tests. This dataset is generated by leveraging the VMAF metric to provide ranking-based annotations, effectively circumventing the data scarcity problem.
Datasets
- Training Data Generation: A novel, large-scale dataset was created using 230 source sequences compressed with four standard codecs (H.264/AVC, H.265/HEVC, AV1, H.266/VVC) at multiple quantization levels. Additional data was generated through resolution adaptation. This resulted in 14,720 distorted sequences, which were then processed to create approximately 204,800 groups of dodecuplet patches with VMAF ranking-based annotations.
- Stage 2 Training Data: Two publicly available VQA databases were used: VMAF+ and IVP.
- Evaluation Datasets: Eight commonly used HD video quality datasets were used for evaluation: NFLX, NFLX-P, BVI-HD, BVI-CCHD, BVI-CCHDDO, MCL-V, SHVC, and VQEGHD3.
Key Insights and Ideas
- Ranking is more important than absolute scores: The paper argues that for VQA, accurately ranking the quality of different distorted videos is more critical than predicting an exact absolute score. This insight forms the basis of their ranking-inspired training.
- Automated dataset creation is feasible: By using VMAF for ranking-based annotations, the authors demonstrate that large-scale VQA datasets can be generated without expensive and time-consuming subjective testing.
- Combining multiple datasets is possible with ranking: The ranking-inspired training strategy in Stage 2 allows for the effective integration of data from different VQA databases, leading to improved model generalization.
- Transformer architecture for patch-level quality: The use of a Swin Transformer in Stage 1 effectively captures local spatio-temporal information within video patches for quality assessment.
Key Results
- State-of-the-art performance: FR-RankDVQA achieved an average SROCC of 0.8972 across eight test sets, outperforming VMAF and other benchmark methods. NR-RankDVQA achieved an average SROCC of 0.7791.
- Superior generalization: The proposed methods demonstrated significantly better performance without re-training or cross-validation on the test databases, indicating strong generalization capabilities.
- FR-RankDVQA outperforms VMAF: For the first time, a deep FR VQA method consistently outperforms VMAF on various video quality databases without re-training.
- Significance: Statistical tests confirmed that FR-RankDVQA's performance was significantly superior to VMAF on several datasets.
Limitations and Future Work
- Runtime complexity: FR-RankDVQA is noted to be slower than VMAF (3.88 times slower), similar to other deep VQA methods.
- Future Work: The authors suggest focusing on more sophisticated pooling network architectures and complexity reduction for future research.
Summary updated