Priorformer: A UGC-VQA Method with content and distortion priors

Yajing Pei, Shiyu Huang, Yiting Lu, Xin Li, Zhibo Chen

Abstract

User Generated Content (UGC) videos are susceptible to complicated and variant degradations and contents, which prevents the existing blind video quality assessment (BVQA) models from good performance since the lack of the adaptability of distortions and contents. To mitigate this, we propose a novel prior-augmented perceptual vision transformer (PriorFormer) for the BVQA of UGC, which boosts its adaptability and representation capability for divergent contents and distortions. Concretely, we introduce two powerful priors, i.e., the content and distortion priors, by extracting the content and distortion embeddings from two pre-trained feature extractors. Then we adopt these two powerful embeddings as the adaptive prior tokens, which are transferred to the vision transformer backbone jointly with implicit quality features. Based on the above strategy, the proposed PriorFormer achieves state-of-the-art performance on three public UGC VQA datasets including KoNViD-1K, LIVE-VQC and YouTube-UGC.

Introduction

Assessing the quality of User-Generated Content (UGC)—the vast volume of video uploaded daily to platforms like YouTube and TikTok—presents a significant challenge for traditional algorithms. Unlike professional cinematography, UGC is characterized by its "in-the-wild" nature, featuring diverse subjects ranging from high-motion gaming to low-light vlogs. These videos often suffer from complex, overlapping distortions such as motion blur, compression artifacts, and sensor noise.

Existing Blind Video Quality Assessment (BVQA) models frequently struggle with these variations because they lack adaptability. Human perception of quality is inherently contextual; for instance, a viewer might tolerate grain in a fast-paced action sequence but find it unacceptable in a static nature shot. Most current models fail to account for this intrinsic relationship between a video's semantic content, its specific distortions, and the final perceived quality. To address this, we introduce PriorFormer, a framework that explicitly incorporates content and distortion priors to mimic the nuanced way humans perceive video quality.

Methodology

The PriorFormer architecture is built upon a prior-augmented perceptual vision transformer designed to integrate explicit contextual information into the quality estimation process. The framework consists of three primary components:

1. Dual Prior Extraction

Rather than relying on a single feature extractor, PriorFormer utilizes two specialized "expert" models to provide context:

  • Content Prior: This module leverages CLIP (Contrastive Language-Image Pre-training), a large-scale vision-language model. By using CLIP, the system gains a fine-grained semantic understanding of the scene, tapping into a model trained on 400 million image-text pairs to provide a "common sense" understanding of what is being depicted.
  • Distortion Prior: This module employs a Distortion Graph Representation (DGR) to identify specific types and levels of degradation. This prior is pre-trained on massive synthetic datasets (Kadid-10k and Kadis-700k) to recognize various artifacts like Gaussian blur or additive noise.

2. Prior-Augmented Transformer

These extracted priors are converted into adaptive "prior tokens." These tokens are fed into a Vision Transformer (ViT) backbone alongside standard implicit quality features (extracted via a ResNet-50). This allows the Transformer’s self-attention mechanism to dynamically weigh quality features based on the specific content and distortion context of the frame.

3. Subjectively-Inspired Temporal Fusion

To aggregate frame-level assessments into a final video-level score, PriorFormer utilizes a Gated Recurrent Unit (GRU) followed by a specialized pooling layer. This layer is designed to mimic "temporal hysteresis"—the human psychological tendency to react sharply to sudden quality drops while remaining skeptical even after quality improves. The model employs Minimal Pooling and Softmin-Weighted Average Pooling to ensure that the "worst" parts of a video contribute significantly to the final score, reflecting human memory biases.

Experimental Results

PriorFormer was evaluated on three gold-standard UGC datasets: KoNViD-1k, LIVE-VQC, and YouTube-UGC (which includes resolutions up to 4K). The model achieved state-of-the-art (SOTA) results across all benchmarks, outperforming both traditional hand-crafted feature methods and modern deep-learning approaches.

Dataset Metric PriorFormer (Proposed) Previous Best (RAPIQUE/VSFA/PVQ)
KoNViD-1k SRCC 0.8291 0.8031
LIVE-VQC SRCC 0.7966 0.7700
YouTube-UGC SRCC 0.8394 0.7873

Note: SRCC (Spearman Rank Order Correlation Coefficient) measures the alignment between model rankings and human subjective scores, where 1.0 represents perfect correlation.

Discussion and Insights

Ablation studies highlight that the Content Prior (CLIP) is particularly vital; its removal resulted in a more significant performance degradation than the removal of the distortion prior. This suggests that semantic context—knowing what is in the frame—is perhaps the most critical factor in judging UGC quality.

However, the model does face limitations regarding the Distortion Prior. Because this component was trained primarily on synthetic distortions, it may have limited capability when encountering the highly complex, hybrid distortions found in real-world UGC. Future research will focus on training distortion extractors on more diverse, real-world datasets and exploring the integration of additional metadata, such as camera settings, as further priors.

Conclusion

PriorFormer demonstrates that explicitly modeling "context" through content and distortion priors significantly improves the accuracy of video quality assessment. By combining foundation models like CLIP with subjectively-inspired temporal fusion, PriorFormer provides a more human-centric approach to evaluating the billions of UGC videos that define the modern digital landscape.

Summary updated

Abstract

User Generated Content (UGC) videos are susceptible to complicated and variant degradations and contents, which prevents the existing blind video quality assessment (BVQA) models from good performance since the lack of the adapability of distortions and contents. To mitigate this, we propose a novel prior-augmented perceptual vision transformer (PriorFormer) for the BVQA of UGC, which boots its adaptability and representation capability for divergent contents and distortions. Concretely, we introduce two powerful priors, i.e., the content and distortion priors, by extracting the content and distortion embeddings from two pre-trained feature extractors. Then we adopt these two powerful embeddings as the adaptive prior tokens, which are transferred to the vision transformer backbone jointly with implicit quality features. Based on the above strategy, the proposed PriorFormer achieves state-of-the-art performance on three public UGC VQA datasets including KoNViD-1K, LIVE-VQC and YouTube-UGC.
BibTeX
@inproceedings{pei2024priorformer,
  author = {Yajing Pei and Shiyu Huang and Yiting Lu and Xin Li and Zhibo Chen},
  title = {{Priorformer: A UGC-VQA Method with content and distortion priors}},
  booktitle = {ICIP},
  year = {2024},
  url = {https://openreview.net/forum?id=NIjijvAskS},
}
Figure