SB-VQA: A Stack-Based Video Quality Assessment Framework for Video Enhancement

Ding-Jiun Huang, Yu-Ting Kao, Tieh-Hung Chuang, Ya-Chun Tsai, Jing-Kai Lou, Shuen-Huei GuanWorkshops @ CVPR 2023 · DOI · arXiv

Introduction

Video Quality Assessment (VQA) has traditionally focused on User-Generated Content (UGC), characterized by natural distortions like camera shake and poor lighting. While existing models excel at identifying these artifacts, they often falter when evaluating enhanced videos—content processed by AI for Super-Resolution (VSR), deblurring, or color correction. These enhancements introduce subtle, non-natural artifacts that metrics like PSNR or SSIM fail to capture. Furthermore, Professionally Generated Content (PGC), such as movie trailers, presents a unique challenge where human perception is influenced by both visual quality and content appeal.

The SB-VQA Framework

The SB-VQA (Stack-Based Video Quality Assessment) framework addresses these challenges through three primary architectural innovations designed to mimic human visual perception while maintaining computational efficiency.

The SB-VQA framework architecture, featuring Grid Mini-patch Sampling, the FANet feature extractor, and the dual-branch patch weighting system.

1. Grid Mini-patch Sampling (GMS)

To process high-resolution video efficiently, SB-VQA employs Grid Mini-patch Sampling. By sampling small, uniform patches across the frame, the model preserves local textures and global structural information without the prohibitive computational cost of processing full-resolution frames.

2. Stacked Ensemble Strategy

To mitigate data-specific bias, the framework utilizes a "stacking" strategy. $K$ versions of the feature extractor are trained on different data folds. Their predictions are then fused using an Extreme Gradient Boosting (XGBoost) regression block, resulting in a more robust and generalized Mean Opinion Score (MOS).

3. Dual-Branch Patch Weighting

The "secret sauce" of SB-VQA is its dual-branch system for patch evaluation:

  • Scoring Branch: Predicts the objective quality of a specific patch.
  • Weighting Branch: Estimates the subjective importance of that patch to a human viewer (e.g., prioritizing a central subject over a background).

The final quality score is a weighted sum of these branches, effectively modeling human visual attention. The core extractor, FANet (Fragment Attention Network), utilizes Swin Transformer blocks to capture complex spatial and temporal dynamics.

Datasets and Benchmarking

The model was evaluated on standard benchmarks like VDPVE (1,211 enhanced videos) and large-scale UGC datasets (LSVQ, KoNViD-1k). Additionally, the authors introduced PGCVQ, a new dataset comprising 1,200 movie trailer clips. PGCVQ includes videos at multiple resolutions (360p to 1080p) and bitrates, paired with YouTube Heatmap data to correlate visual quality with viewer interest.

Key Results and Discussion

SB-VQA achieved state-of-the-art performance on the NTIRE 2023 Video Quality Assessment Challenge, reaching a main score of 0.7635 on the VDPVE test set, significantly outperforming the previous SOTA, Fast-VQA (0.7330).

Ablation studies confirmed the efficacy of the framework's components. The dual-branch weighting system improved the Spearman Rank Correlation (SRCC) from 0.820 to 0.838, while the full stacked architecture further boosted performance to 0.862.

Predicted Mean Opinion Scores (MOS) across various resolutions and bitrates for the PGCVQ dataset, demonstrating the model's consistency.

Testing on the PGCVQ dataset demonstrated that the model's predicted MOS consistently scales with increases in bitrate and resolution, confirming its reliability for professional streaming applications. Interestingly, the study found a strong correlation between "quality" and "interest" peaks in YouTube heatmaps, suggesting that action-heavy or visually striking scenes drive both higher engagement and higher perceived quality.

Conclusion and Future Directions

SB-VQA represents a significant step forward in assessing AI-enhanced and professional video content. While highly effective, the model faces challenges in cross-dataset generalization, where the regression block may overfit to specific distributions. Future work aims to integrate semantic understanding—perhaps through Large Language Models (LLMs) or audio analysis—to better account for the "story" or context of a scene, which plays a crucial role in human quality perception for PGC.

Summary updated

BibTeX
@inproceedings{huang2023sb,
  author = {Ding-Jiun Huang and Yu-Ting Kao and Tieh-Hung Chuang and Ya-Chun Tsai and Jing-Kai Lou and Shuen-Huei Guan},
  title = {{SB-VQA: A Stack-Based Video Quality Assessment Framework for Video Enhancement}},
  booktitle = {Workshops @ CVPR},
  year = {2023},
  doi = {10.1109/cvprw59228.2023.00163},
}
Figure