Abstract
Video Quality Assessment (VQA) is increasingly critical as streaming platforms like TikTok and Kwai dominate digital consumption. Unlike professionally produced content, User-Generated Content (UGC) presents unique challenges: a few highly distorted frames can disproportionately impact perceived quality, and multiple types of distortions often coexist. We present the Visual Quality Transformer (VQT), which utilizes Sparse Temporal Attention (STA) and a Multi-Pathway Temporal Network (MPTN) to efficiently capture these quality-related features. VQT achieves state-of-the-art performance on major benchmarks and outperforms industrial standards like VMAF.
Introduction
The rapid expansion of streaming media has made Video Quality Assessment (VQA) a vital tool for maintaining user experience. While VQA for professionally produced content (PGC) is well-established, User-Generated Content (UGC) remains a "Wild West" of unpredictable distortions. Traditional sequence-based models often treat all frames with equal importance, which fails to align with human perception in the context of UGC.
The Challenges of UGC Quality
The researchers identify two primary hurdles that existing VQA models struggle to overcome:
- The Bottleneck Effect: Human perception is highly sensitive to "bad apples." A few seconds of severe blurriness or blocking artifacts can ruin the experience of an otherwise high-quality video. Most models use simple averaging, which dilutes the impact of these critical distortions.
- Co-existing Distortions: UGC is prone to overlapping issues, such as simultaneous camera shake and heavy compression. These distortions occur at different temporal scales and frequencies, requiring a more nuanced sampling strategy than a standard uniform approach.
The Visual Quality Transformer (VQT)
To address these issues, the Visual Quality Transformer (VQT) introduces two architectural innovations designed to extract quality-related features more efficiently.
Sparse Temporal Attention (STA)
Standard attention mechanisms are computationally expensive, scaling at $O(T^2)$. VQT introduces Sparse Temporal Attention (STA), which reduces this complexity to $O(T \log T)$. By calculating the KL divergence between a frame's correlation distribution and a uniform distribution, STA identifies "informative" frames—those that deviate significantly and likely contain unique distortions. This allows the model to focus its computational budget on the frames that matter most to human viewers.
Multi-Pathway Temporal Network (MPTN)
Because different distortions (e.g., a momentary glitch vs. persistent motion blur) require different "zoom levels," the MPTN utilizes multiple STA modules in parallel. Each pathway operates with a different degree of sparsity. This multi-scale approach allows the model to capture short-lived artifacts and long-term quality trends simultaneously, providing a holistic view of the video's perceptual quality.
Experimental Results
VQT was evaluated across several No-Reference (NR) and Full-Reference (FR) datasets, demonstrating significant improvements over existing methods.
Performance Benchmarks
On NR-VQA datasets like KoNViD-1k and YouTube-UGC, VQT achieved state-of-the-art results, improving the Pearson Linear Correlation Coefficient (PLCC) by over 2% compared to previous leaders. Furthermore, in cross-dataset evaluations, VQT outperformed widely-adopted industrial algorithms, including Netflix’s VMAF and Apple’s AVQT, suggesting superior generalization to diverse video formats.
Efficiency and Versatility
Beyond accuracy, VQT offers practical advantages for production environments:
- Computational Efficiency: VQT reduces TFLOPs by 22% and increases inference speed by 13% compared to the standard TimeSformer. It can process a 30s video in approximately 0.5s on a GPU.
- Generalization: When tested on the Kinetics-400 action recognition dataset, VQT-based backbones outperformed established models like Video Swin Transformer, indicating that quality-aware features are broadly beneficial for general video understanding tasks.
Conclusion
The Visual Quality Transformer represents a significant step forward in aligning machine vision with human perception for UGC. By prioritizing "bottleneck" frames and addressing multi-distortion distributions through sparse, multi-pathway attention, VQT provides a robust and efficient solution for real-world VQA. Future work will focus on adaptive keyframe selection and model compression to further optimize VQT for mobile and real-time applications.
Summary updated