TL;DR
PriorFormer is a novel framework for User-Generated Content (UGC) video quality assessment that explicitly incorporates "content" and "distortion" priors into a Vision Transformer. By understanding what is in a video and how it is degraded, the model achieves state-of-the-art performance across major benchmarks, mimicking the nuanced way humans perceive video quality.
What Problem Does This Paper Address?
Assessing the quality of User-Generated Content (UGC)—the billions of videos uploaded to platforms like YouTube and TikTok—is a nightmare for traditional algorithms. Unlike professional cinema, UGC is "in-the-wild": it features incredibly diverse subjects (from high-octane gaming to shaky sunset vlogs) and complex, overlapping distortions (low light, motion blur, and heavy compression).
Existing Blind Video Quality Assessment (BVQA) models often fail because they lack adaptability. Human perception is subjective; we might forgive a bit of grain in a fast-paced sports clip but find it intolerable in a high-definition nature documentary. Most current models treat all pixels the same, ignoring the intrinsic relationship between the video's content, its specific distortions, and the final perceived quality.
Methods
The authors introduce PriorFormer, a prior-augmented perceptual vision transformer. The architecture is built on three main pillars:
- Dual Prior Extraction: Instead of relying on a single backbone, the model uses two specialized, pre-trained "experts" to extract context:
- Content Prior: Uses CLIP (a large vision-language model) to provide a fine-grained semantic understanding of the scene.
- Distortion Prior: Uses a Distortion Graph Representation (DGR) to identify the specific types and levels of degradation present in the frames.
- Prior-Augmented Transformer: These priors are converted into "tokens" and fed into a Vision Transformer (ViT) alongside standard quality features (extracted via ResNet-50). This allows the Transformer's self-attention mechanism to weigh quality features differently based on the content and distortion context.
- Subjectively-Inspired Temporal Fusion: To turn frame-level scores into a video-level score, the model uses a Gated Recurrent Unit (GRU) followed by a specialized pooling layer. This layer mimics "temporal hysteresis"—the human tendency to react sharply to a sudden drop in quality but remain skeptical even after the quality improves.
Datasets
The model was evaluated on three gold-standard UGC datasets:
- KoNViD-1k: 1,200 "in-the-wild" videos.
- LIVE-VQC: 585 videos captured by 80 different users with 101 different devices.
- YouTube-UGC: 1,380 videos spanning 15 categories (gaming, sports, etc.) with resolutions up to 4K.
Additionally, the distortion prior was pre-trained on the Kadid-10k and Kadis-700k datasets, which contain massive amounts of synthetically distorted images.
Key Insights and Ideas
- Semantic Context is King: Ablation studies showed that removing the Content Prior (CLIP) caused a more significant performance drop than removing the distortion prior. This suggests that "knowing what you are looking at" is perhaps the most critical factor in judging UGC quality.
- The Power of Foundation Models: By leveraging CLIP, the researchers tapped into a model trained on 400 million image-text pairs, giving PriorFormer a "common sense" understanding of video content that standard ImageNet-trained models lack.
- Modeling Human Memory: The inclusion of "Minimal Pooling" and "Softmin-Weighted Average Pooling" acknowledges that humans don't just average out their experience; we remember the "worst" parts of a video more vividly than the "average" parts.
Key Results
PriorFormer achieved state-of-the-art (SOTA) results across all three benchmarks, outperforming both traditional hand-crafted feature methods and modern deep-learning approaches.
| Dataset | Metric | PriorFormer (Proposed) | Previous Best (RAPIQUE/VSFA/PVQ) |
|---|---|---|---|
| KoNViD-1k | SRCC | 0.8291 | 0.8031 |
| LIVE-VQC | SRCC | 0.7966 | 0.7700 |
| YouTube-UGC | SRCC | 0.8394 | 0.7873 |
Note: SRCC (Spearman Rank Order Correlation Coefficient) measures how well the model's rankings align with human rankings (closer to 1.0 is better).
Limitations and Future Work
The authors note a slight limitation in the Distortion Prior: because it was trained on synthetic distortions (like Gaussian blur or additive noise), it has a "limited capability" to fully grasp the messy, hybrid distortions found in real-world UGC.
Future work will likely focus on:
- Training distortion extractors on more diverse, real-world datasets.
- Exploring even deeper integration of UGC-specific metadata (like camera type or upload settings) as additional priors.
Summary updated