Abstract
User Generated Content (UGC) videos are susceptible to complicated and variant degradations and contents, which prevents the existing blind video quality assessment (BVQA) models from good performance since the lack of the adaptability of distortions and contents. To mitigate this, we propose a novel prior-augmented perceptual vision transformer (PriorFormer) for the BVQA of UGC, which boosts its adaptability and representation capability for divergent contents and distortions. Concretely, we introduce two powerful priors, i.e., the content and distortion priors, by extracting the content and distortion embeddings from two pre-trained feature extractors. Then we adopt these two powerful embeddings as the adaptive prior tokens, which are transferred to the vision transformer backbone jointly with implicit quality features. Based on the above strategy, the proposed PriorFormer achieves state-of-the-art performance on three public UGC VQA datasets including KoNViD-1K, LIVE-VQC and YouTube-UGC.
Introduction
Assessing the quality of User-Generated Content (UGC)—the vast volume of video uploaded daily to platforms like YouTube and TikTok—presents a significant challenge for traditional algorithms. Unlike professional cinematography, UGC is characterized by its "in-the-wild" nature, featuring diverse subjects ranging from high-motion gaming to low-light vlogs. These videos often suffer from complex, overlapping distortions such as motion blur, compression artifacts, and sensor noise.
Existing Blind Video Quality Assessment (BVQA) models frequently struggle with these variations because they lack adaptability. Human perception of quality is inherently contextual; for instance, a viewer might tolerate grain in a fast-paced action sequence but find it unacceptable in a static nature shot. Most current models fail to account for this intrinsic relationship between a video's semantic content, its specific distortions, and the final perceived quality. To address this, we introduce PriorFormer, a framework that explicitly incorporates content and distortion priors to mimic the nuanced way humans perceive video quality.
Methodology
The PriorFormer architecture is built upon a prior-augmented perceptual vision transformer designed to integrate explicit contextual information into the quality estimation process. The framework consists of three primary components:
1. Dual Prior Extraction
Rather than relying on a single feature extractor, PriorFormer utilizes two specialized "expert" models to provide context:
- Content Prior: This module leverages CLIP (Contrastive Language-Image Pre-training), a large-scale vision-language model. By using CLIP, the system gains a fine-grained semantic understanding of the scene, tapping into a model trained on 400 million image-text pairs to provide a "common sense" understanding of what is being depicted.
- Distortion Prior: This module employs a Distortion Graph Representation (DGR) to identify specific types and levels of degradation. This prior is pre-trained on massive synthetic datasets (Kadid-10k and Kadis-700k) to recognize various artifacts like Gaussian blur or additive noise.
2. Prior-Augmented Transformer
These extracted priors are converted into adaptive "prior tokens." These tokens are fed into a Vision Transformer (ViT) backbone alongside standard implicit quality features (extracted via a ResNet-50). This allows the Transformer’s self-attention mechanism to dynamically weigh quality features based on the specific content and distortion context of the frame.
3. Subjectively-Inspired Temporal Fusion
To aggregate frame-level assessments into a final video-level score, PriorFormer utilizes a Gated Recurrent Unit (GRU) followed by a specialized pooling layer. This layer is designed to mimic "temporal hysteresis"—the human psychological tendency to react sharply to sudden quality drops while remaining skeptical even after quality improves. The model employs Minimal Pooling and Softmin-Weighted Average Pooling to ensure that the "worst" parts of a video contribute significantly to the final score, reflecting human memory biases.
Experimental Results
PriorFormer was evaluated on three gold-standard UGC datasets: KoNViD-1k, LIVE-VQC, and YouTube-UGC (which includes resolutions up to 4K). The model achieved state-of-the-art (SOTA) results across all benchmarks, outperforming both traditional hand-crafted feature methods and modern deep-learning approaches.
| Dataset | Metric | PriorFormer (Proposed) | Previous Best (RAPIQUE/VSFA/PVQ) |
|---|---|---|---|
| KoNViD-1k | SRCC | 0.8291 | 0.8031 |
| LIVE-VQC | SRCC | 0.7966 | 0.7700 |
| YouTube-UGC | SRCC | 0.8394 | 0.7873 |
Note: SRCC (Spearman Rank Order Correlation Coefficient) measures the alignment between model rankings and human subjective scores, where 1.0 represents perfect correlation.
Discussion and Insights
Ablation studies highlight that the Content Prior (CLIP) is particularly vital; its removal resulted in a more significant performance degradation than the removal of the distortion prior. This suggests that semantic context—knowing what is in the frame—is perhaps the most critical factor in judging UGC quality.
However, the model does face limitations regarding the Distortion Prior. Because this component was trained primarily on synthetic distortions, it may have limited capability when encountering the highly complex, hybrid distortions found in real-world UGC. Future research will focus on training distortion extractors on more diverse, real-world datasets and exploring the integration of additional metadata, such as camera settings, as further priors.
Conclusion
PriorFormer demonstrates that explicitly modeling "context" through content and distortion priors significantly improves the accuracy of video quality assessment. By combining foundation models like CLIP with subjectively-inspired temporal fusion, PriorFormer provides a more human-centric approach to evaluating the billions of UGC videos that define the modern digital landscape.
Summary updated