Capturing Co-existing Distortions in User-Generated Content for No-reference Video Quality Assessment

Kun Yuan, Zishang Kong, Chuanchuan Zheng, Ming Sun, Xing Wen

Abstract

Video Quality Assessment (VQA) is increasingly critical as streaming platforms like TikTok and Kwai dominate digital consumption. Unlike professionally produced content, User-Generated Content (UGC) presents unique challenges: a few highly distorted frames can disproportionately impact perceived quality, and multiple types of distortions often coexist. We present the Visual Quality Transformer (VQT), which utilizes Sparse Temporal Attention (STA) and a Multi-Pathway Temporal Network (MPTN) to efficiently capture these quality-related features. VQT achieves state-of-the-art performance on major benchmarks and outperforms industrial standards like VMAF.

Introduction

The rapid expansion of streaming media has made Video Quality Assessment (VQA) a vital tool for maintaining user experience. While VQA for professionally produced content (PGC) is well-established, User-Generated Content (UGC) remains a "Wild West" of unpredictable distortions. Traditional sequence-based models often treat all frames with equal importance, which fails to align with human perception in the context of UGC.

The Challenges of UGC Quality

The researchers identify two primary hurdles that existing VQA models struggle to overcome:

  1. The Bottleneck Effect: Human perception is highly sensitive to "bad apples." A few seconds of severe blurriness or blocking artifacts can ruin the experience of an otherwise high-quality video. Most models use simple averaging, which dilutes the impact of these critical distortions.
  2. Co-existing Distortions: UGC is prone to overlapping issues, such as simultaneous camera shake and heavy compression. These distortions occur at different temporal scales and frequencies, requiring a more nuanced sampling strategy than a standard uniform approach.

The Visual Quality Transformer (VQT)

To address these issues, the Visual Quality Transformer (VQT) introduces two architectural innovations designed to extract quality-related features more efficiently.

Sparse Temporal Attention (STA)

Standard attention mechanisms are computationally expensive, scaling at $O(T^2)$. VQT introduces Sparse Temporal Attention (STA), which reduces this complexity to $O(T \log T)$. By calculating the KL divergence between a frame's correlation distribution and a uniform distribution, STA identifies "informative" frames—those that deviate significantly and likely contain unique distortions. This allows the model to focus its computational budget on the frames that matter most to human viewers.

Multi-Pathway Temporal Network (MPTN)

Because different distortions (e.g., a momentary glitch vs. persistent motion blur) require different "zoom levels," the MPTN utilizes multiple STA modules in parallel. Each pathway operates with a different degree of sparsity. This multi-scale approach allows the model to capture short-lived artifacts and long-term quality trends simultaneously, providing a holistic view of the video's perceptual quality.

Experimental Results

VQT was evaluated across several No-Reference (NR) and Full-Reference (FR) datasets, demonstrating significant improvements over existing methods.

Performance Benchmarks

On NR-VQA datasets like KoNViD-1k and YouTube-UGC, VQT achieved state-of-the-art results, improving the Pearson Linear Correlation Coefficient (PLCC) by over 2% compared to previous leaders. Furthermore, in cross-dataset evaluations, VQT outperformed widely-adopted industrial algorithms, including Netflix’s VMAF and Apple’s AVQT, suggesting superior generalization to diverse video formats.

Efficiency and Versatility

Beyond accuracy, VQT offers practical advantages for production environments:

  • Computational Efficiency: VQT reduces TFLOPs by 22% and increases inference speed by 13% compared to the standard TimeSformer. It can process a 30s video in approximately 0.5s on a GPU.
  • Generalization: When tested on the Kinetics-400 action recognition dataset, VQT-based backbones outperformed established models like Video Swin Transformer, indicating that quality-aware features are broadly beneficial for general video understanding tasks.

Conclusion

The Visual Quality Transformer represents a significant step forward in aligning machine vision with human perception for UGC. By prioritizing "bottleneck" frames and addressing multi-distortion distributions through sparse, multi-pathway attention, VQT provides a robust and efficient solution for real-world VQA. Future work will focus on adaptive keyframe selection and model compression to further optimize VQT for mobile and real-time applications.

Summary updated

Abstract

Video Quality Assessment (VQA), which aims to predict the perceptual quality of a video, has attracted raising attention with the rapid development of streaming media technology, such as Facebook, TikTok, Kwai, and so on. Compared with other sequence-based visual tasks (\textit{e.g.,} action recognition), VQA faces two under-estimated challenges unresolved in User Generated Content (UGC) videos. \textit{First}, it is not rare that several frames containing serious distortions (\textit{e.g.,}blocking, blurriness), can determine the perceptual quality of the whole video, while other sequence-based tasks require more frames of equal importance for representations. \textit{Second}, the perceptual quality of a video exhibits a multi-distortion distribution, due to the differences in the duration and probability of occurrence for various distortions. In order to solve the above challenges, we propose \textit{Visual Quality Transformer (VQT)} to extract quality-related sparse features more efficiently. Methodologically, a Sparse Temporal Attention (STA) is proposed to sample keyframes by analyzing the temporal correlation between frames, which reduces the computational complexity from $O(T^2)$ to $O(T \log T)$. Structurally, a Multi-Pathway Temporal Network (MPTN) utilizes multiple STA modules with different degrees of sparsity in parallel, capturing co-existing distortions in a video. Experimentally, VQT demonstrates superior performance than many \textit{state-of-the-art} methods in three public no-reference VQA datasets. Furthermore, VQT shows better performance in four full-reference VQA datasets against widely-adopted industrial algorithms (\textit{i.e.,} VMAF and AVQT).
BibTeX
@inproceedings{yuan2023capturing,
  author = {Kun Yuan and Zishang Kong and Chuanchuan Zheng and Ming Sun and Xing Wen},
  title = {{Capturing Co-existing Distortions in User-Generated Content for No-reference Video Quality Assessment}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=Y-_r6JQ6vux},
}
Figure