TL;DR
This paper tackles the challenge of assessing video quality in user-generated content (UGC) without requiring extensive human annotations. They introduce a novel semi-supervised learning framework that leverages a self-supervised pre-trained feature extractor and a knowledge transfer mechanism between two quality assessment models to achieve state-of-the-art performance with limited labeled data.
What Problem Does This Paper Address?
User-generated videos (UGC) are ubiquitous, but assessing their perceptual quality is a significant challenge. Traditional methods often require large datasets of human-annotated videos for training, which are expensive and time-consuming to create. Furthermore, UGC videos often suffer from complex, mixed distortions that are difficult to model. This paper aims to develop a robust video quality assessment (VQA) model that can achieve excellent generalization performance on various datasets, even when trained with very limited labeled data, by effectively utilizing unlabeled data.
Methods
The proposed approach consists of two main stages:
- Spatio-Temporal Visual Quality Representation Learning (ST-VQRL): This is a self-supervised pre-training phase. The authors design a framework to learn robust quality-aware features from unlabeled videos. They utilize a transformer-based architecture (specifically, Video Swin-T) and a novel "statistical contrastive loss." This loss goes beyond simple point-wise similarity by considering both global and local variations in spatio-temporal features, making the learning more robust. They also introduce "Quality Consistent Sampling" (QCS) to generate augmented video clips with similar content but different distortions, which are used as positive pairs in the contrastive learning.
- Semi-Supervised Learning for VQA (SSL-VQA): This stage leverages the pre-trained ST-VQRL backbone and combines limited labeled data with a large amount of unlabeled data. They employ a dual-model learning setup:
- Regressor-based Quality Model: Directly maps video features to a quality score.
-
Distance-based Quality Model: Maps the distance between a distorted video's features and a corpus of pristine video features to a quality score.
Knowledge transfer is crucial here. They use "intra-model consistency loss" to ensure stable predictions for augmented clips within each model. More importantly, they implement a "knowledge transfer loss" that intelligently transfers knowledge between the two models. This transfer is conditional: if one model's prediction is stable (indicated by low intra-model consistency error), its prediction is used as a pseudo-label for the other model. This prevents erroneous predictions from one model from negatively impacting the other.
Datasets
The paper uses several datasets for training and evaluation:
- LIVE-FB Large-Scale Social Video Quality (LSVQ): Used for generating synthetically distorted UGC videos for self-supervised pre-training and as a primary dataset for evaluation.
- LIVE-VQA, LIVE Mobile, CSIQ VQD, EPFL-POLIMI, ECCV-EVVQ: Used as a corpus of pristine videos for the distance-based quality model.
- KoNVid-1K, LIVE VQC, YouTube-UGC, LIVE Qualcomm: Used for evaluating the model's performance on different types of UGC videos and in cross-database settings.
No new datasets were introduced.
Key Insights and Ideas
- Self-supervised learning is key for limited data: The paper strongly advocates for self-supervised pre-training to extract rich, quality-aware features that are essential for effective VQA with minimal human labels.
- Statistical contrastive loss for richer features: Moving beyond simple similarity measures in contrastive learning by incorporating statistical properties of features leads to more robust and perceptually relevant representations.
- Dual-model knowledge transfer for robustness: Using two different VQA models (regressor-based and distance-based) and enabling them to learn from each other, especially when one is more confident, is a powerful way to leverage unlabeled data and improve generalization.
- Transformer architectures are suitable for VQA: The use of Video Swin-T demonstrates the effectiveness of transformer models in capturing both spatial and temporal distortions in videos.
Key Results
- State-of-the-art performance with limited data: The proposed SSL-VQA method achieves significant improvements over existing methods when trained with limited labeled data. It improves performance by around 10% when only limited labeled data is used and by around 15% when unlabeled data is also utilized in the SSL framework.
- Strong cross-database performance: The model demonstrates excellent generalization capabilities, performing well on datasets it was not explicitly trained on.
- Superiority over classical and supervised methods: The paper shows that their SSL-VQA method outperforms classical feature-based methods and supervised deep learning methods when trained with limited labeled data.
- Effectiveness of components: Ablation studies confirm the benefits of the statistical contrastive loss, intra-model consistency, and knowledge transfer loss.
Limitations and Future Work
- Computational Intensity: Training transformer-based models on high-resolution videos is computationally demanding, although the authors employ strategies like QCS to mitigate this.
- Reliance on Pristine Videos: The distance-based model relies on a corpus of pristine videos, which might not always be readily available or representative of all possible clean video characteristics.
- Future Work: The paper suggests exploring more advanced SSL techniques and investigating explainability for VQA models. The authors also highlight the potential for their approach to make deep learning-based VQA more accessible by reducing the reliance on large labeled datasets.
Summary updated