Capturing Co-existing Distortions in User-Generated Content for No-reference Video Quality Assessment

Kun Yuan, Zishang Kong, Chuanchuan Zheng, Ming Sun, Xing Wen

TL;DR

Researchers from Kuaishou and Peking University have introduced the Visual Quality Transformer (VQT), a novel framework designed to assess the quality of messy, real-world user-generated content (UGC). By using a "Sparse Temporal Attention" mechanism that identifies key distorted frames and a multi-pathway network to capture diverse types of glitches, VQT outperforms current state-of-the-art models while being significantly more computationally efficient.

What Problem Does This Paper Address?

Video Quality Assessment (VQA) is the science of teaching computers to "see" video quality the way humans do. This is relatively easy for professionally produced content (PGC), but User-Generated Content (UGC)—the kind you see on TikTok, Kwai, or YouTube—is the "Wild West."

The authors identify two major hurdles in UGC quality assessment:

  1. The Bottleneck Effect: In human perception, a few seconds of terrible blocking or blurriness can ruin the experience of an otherwise perfect video. Most existing models use simple arithmetic averages of frame quality, which fails to capture how these "bad apples" dominate our judgment.
  2. Co-existing Distortions: UGC videos often suffer from multiple, overlapping issues—like camera shake (temporal) happening at the same time as heavy compression artifacts (spatial). These distortions occur at different frequencies and durations, making them hard to capture with a "one-size-fits-all" sampling strategy.

Methods

The authors propose the Visual Quality Transformer (VQT), which introduces two clever architectural innovations:

  • Sparse Temporal Attention (STA): Instead of looking at every single frame (which is computationally expensive, $O(T^2)$), STA uses a selection strategy based on the Johnson-Lindenstrauss lemma. It calculates the KL divergence between a frame's correlation distribution and a uniform distribution. Essentially, it identifies "informative" frames—those that stand out because they contain unique distortions—and focuses its attention there. This reduces complexity to $O(T \log T)$.
  • Multi-Pathway Temporal Network (MPTN): Since different distortions (like a quick flash of "blocking" vs. a long "motion blur") require different temporal scales to detect, MPTN stacks multiple STA modules in parallel. Each pathway uses a different degree of sparsity, allowing the model to "zoom in" on short-lived glitches and "zoom out" to see long-term quality trends simultaneously.

Datasets

The model was rigorously tested across several benchmarks:

  • No-Reference (NR) Datasets: KoNViD-1k, LIVE-VQC, and YouTube-UGC. These are "in-the-wild" videos where no "perfect" original version exists for comparison.
  • Full-Reference (FR) Datasets: ICME-FR, VQEG HD3, NFLX, and Waterloo IVC 4k. These were used to test the model's generalization against industrial standards like Netflix’s VMAF.
  • Generalization: The authors also tested VQT on Kinetics-400, a massive video action recognition dataset, to see if their quality-focused transformer could handle general computer vision tasks.

Key Insights and Ideas

  • Sparsity as a Feature, Not a Bug: Usually, sparse sampling is a compromise for speed. Here, the authors argue that sparse sampling—if done intelligently—actually improves accuracy by filtering out redundant, "boring" frames and highlighting the distorted ones that humans actually care about.
  • The "Bad Frame" Priority: The results confirm that frames with the lowest quality scores have a disproportionate impact on the final MOS (Mean Opinion Score). VQT’s attention maps show it naturally gravitates toward these "quality bottlenecks."
  • Plug-and-Play Generalization: Interestingly, the VQT module isn't just for quality. When plugged into a standard video classification backbone, it outperformed established models like TimeSformer and Video Swin Transformer on Kinetics-400, suggesting that "quality-aware" features are useful for understanding video content in general.

Key Results

  • Superior Accuracy: VQT achieved state-of-the-art results on all three major NR-VQA datasets. For example, it improved the PLCC (Pearson Linear Correlation Coefficient) by 2.14% on KoNViD-1k and 2.17% on YouTube-UGC over previous best methods.
  • Beating Industrial Giants: In cross-dataset evaluations, VQT outperformed widely-used industrial algorithms like VMAF and Apple’s AVQT, proving it generalizes better to new, unseen video formats.
  • Efficiency Gains: Compared to the standard TimeSformer, VQT reduced computational costs (TFLOPs) by 22% and increased inference speed by 13%. With an inference time of 0.5s for a 30s video on a GPU, it is fast enough for real-time monitoring in production environments.

Limitations and Future Work

The authors acknowledge that the selection of keyframes in the STA module currently relies on predefined hyperparameters (like the number of frames to sample). While effective, this isn't fully "smart" yet.

Future Direction: They suggest moving toward adaptive keyframe selection, where the model learns to decide exactly how many and which frames to sample based on the specific content of the video, potentially using a reinforcement learning approach. They also mention exploring model compression techniques like knowledge distillation to make VQT even lighter for mobile deployment.

Summary updated

Abstract

Video Quality Assessment (VQA), which aims to predict the perceptual quality of a video, has attracted raising attention with the rapid development of streaming media technology, such as Facebook, TikTok, Kwai, and so on. Compared with other sequence-based visual tasks (e.g., action recognition), VQA faces two under-estimated challenges unresolved in User Generated Content (UGC) videos. First, it is not rare that several frames containing serious distortions (e.g., blocking, blurriness), can determine the perceptual quality of the whole video, while other sequence-based tasks require more frames of equal importance for representations.Second, the perceptual quality of a video exhibits a multi-distortion distribution, due to the differences in the duration and probability of occurrence for various distortions. In order to solve the above challenges, we propose Visual Quality Transformer (VQT) to extract quality-related sparse features more efficiently. Methodologically, a Sparse Temporal Attention (STA) is proposed to sample keyframes by analyzing the temporal correlation between frames, which reduces the computational complexity from O(T2) to O(T log T). Structurally, a Multi-Pathway Temporal Network (MPTN) utilizes multiple STA modules with different degrees of sparsity in parallel, capturing co-existing distortions in a video. Experimentally, VQT demonstrates superior performance than many state-of-the-art methods in three public no-reference VQA datasets. Furthermore, VQT shows better performance in four full-reference VQA datasets against widely-adopted industrial algorithms (e.g., VMAF and AVQT).
BibTeX
@inproceedings{yuan2023capturing,
  author = {Kun Yuan and Zishang Kong and Chuanchuan Zheng and Ming Sun and Xing Wen},
  title = {{Capturing Co-existing Distortions in User-Generated Content for No-reference Video Quality Assessment}},
  booktitle = {ACM Multimedia},
  year = {2023},
  url = {https://openreview.net/forum?id=Y-_r6JQ6vux},
}

Loading related content...

Figure