Introduction
Video Quality Assessment (VQA) has traditionally focused on User-Generated Content (UGC), characterized by natural distortions like camera shake and poor lighting. While existing models excel at identifying these artifacts, they often falter when evaluating enhanced videos—content processed by AI for Super-Resolution (VSR), deblurring, or color correction. These enhancements introduce subtle, non-natural artifacts that metrics like PSNR or SSIM fail to capture. Furthermore, Professionally Generated Content (PGC), such as movie trailers, presents a unique challenge where human perception is influenced by both visual quality and content appeal.
The SB-VQA Framework
The SB-VQA (Stack-Based Video Quality Assessment) framework addresses these challenges through three primary architectural innovations designed to mimic human visual perception while maintaining computational efficiency.
1. Grid Mini-patch Sampling (GMS)
To process high-resolution video efficiently, SB-VQA employs Grid Mini-patch Sampling. By sampling small, uniform patches across the frame, the model preserves local textures and global structural information without the prohibitive computational cost of processing full-resolution frames.
2. Stacked Ensemble Strategy
To mitigate data-specific bias, the framework utilizes a "stacking" strategy. $K$ versions of the feature extractor are trained on different data folds. Their predictions are then fused using an Extreme Gradient Boosting (XGBoost) regression block, resulting in a more robust and generalized Mean Opinion Score (MOS).
3. Dual-Branch Patch Weighting
The "secret sauce" of SB-VQA is its dual-branch system for patch evaluation:
- Scoring Branch: Predicts the objective quality of a specific patch.
- Weighting Branch: Estimates the subjective importance of that patch to a human viewer (e.g., prioritizing a central subject over a background).
The final quality score is a weighted sum of these branches, effectively modeling human visual attention. The core extractor, FANet (Fragment Attention Network), utilizes Swin Transformer blocks to capture complex spatial and temporal dynamics.
Datasets and Benchmarking
The model was evaluated on standard benchmarks like VDPVE (1,211 enhanced videos) and large-scale UGC datasets (LSVQ, KoNViD-1k). Additionally, the authors introduced PGCVQ, a new dataset comprising 1,200 movie trailer clips. PGCVQ includes videos at multiple resolutions (360p to 1080p) and bitrates, paired with YouTube Heatmap data to correlate visual quality with viewer interest.
Key Results and Discussion
SB-VQA achieved state-of-the-art performance on the NTIRE 2023 Video Quality Assessment Challenge, reaching a main score of 0.7635 on the VDPVE test set, significantly outperforming the previous SOTA, Fast-VQA (0.7330).
Ablation studies confirmed the efficacy of the framework's components. The dual-branch weighting system improved the Spearman Rank Correlation (SRCC) from 0.820 to 0.838, while the full stacked architecture further boosted performance to 0.862.
Testing on the PGCVQ dataset demonstrated that the model's predicted MOS consistently scales with increases in bitrate and resolution, confirming its reliability for professional streaming applications. Interestingly, the study found a strong correlation between "quality" and "interest" peaks in YouTube heatmaps, suggesting that action-heavy or visually striking scenes drive both higher engagement and higher perceived quality.
Conclusion and Future Directions
SB-VQA represents a significant step forward in assessing AI-enhanced and professional video content. While highly effective, the model faces challenges in cross-dataset generalization, where the regression block may overfit to specific distributions. Future work aims to integrate semantic understanding—perhaps through Large Language Models (LLMs) or audio analysis—to better account for the "story" or context of a scene, which plays a crucial role in human quality perception for PGC.
Summary updated