UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content

Zhengzhong Tu, Yilin Wang, Neil Birkbeck, Balu Adsumilli, Alan C. BovikIEEE TIP 2020 · DOI · OpenReview

Abstract

Recent years have witnessed an explosion of user-generated content (UGC) videos shared and streamed over the Internet. Accordingly, there is a great need for accurate video quality assessment (VQA) models for UGC/consumer videos to monitor, control, and optimize this vast content. Blind quality prediction of in-the-wild videos is quite challenging, since the quality degradations of UGC content are unpredictable, complicated, and often commingled. Here we contribute to advancing the UGC-VQA problem by conducting a comprehensive evaluation of leading no-reference/blind VQA (BVQA) features and models on a fixed evaluation architecture, yielding new empirical insights on both subjective video quality studies and VQA model design. By employing a feature selection strategy on top of leading VQA model features, we are able to extract 60 of the 763 statistical features used by the leading models to create a new fusion-based BVQA model, which we dub the VIDeo quality EVALuator (VIDEVAL), that effectively balances the trade-off between VQA performance and efficiency. Our experimental results show that VIDEVAL achieves state-of-the-art performance at considerably lower computational cost than other leading models.

Introduction

Video dominates Internet traffic, much of it being UGC from platforms like YouTube, TikTok, and Facebook. Unlike "legacy" video quality assessment (VQA), which focuses on synthetic distortions (like a specific level of compression added to a perfect reference video), UGC is messy. It suffers from "authentic" distortions—blur, noise, shaky camera work, and poor lighting—often all commingled at once. Furthermore, in real-world scenarios, we don't have a "pristine" reference video to compare against. This necessitates Blind Video Quality Assessment (BVQA). Existing models often struggle to generalize across the vast diversity of UGC. This work benchmarks current models and introduces a new, more robust standard for the field.

The VIDEVAL Model

To address the limitations of existing BVQA models, we developed VIDEVAL (VIDeo quality EVALuator). The core innovation is a rigorous feature selection process. We started with a pool of 763 statistical features from top-performing models (including BRISQUE, GM-LOG, and TLVQM). Using a two-step selection strategy involving machine learning rankings (Random Forest and SVM), we distilled this down to a "best-of-the-best" set of just 60 features. This allows VIDEVAL to maintain high accuracy without the massive computational overhead of more complex models.

Datasets and Content Diversity

The study utilizes three major, large-scale UGC datasets that represent the current state of the art:

  • KoNViD-1k: 1,200 public-domain videos sampled from Flickr.
  • LIVE-VQC: 585 videos captured by mobile devices.
  • YouTube-UGC: 1,380 video clips sampled from millions of YouTube videos.

To create a more robust benchmark, we also introduced All-Combined, a composite dataset merging all three after carefully calibrating their subjective Mean Opinion Scores (MOS).

Figure 1: Sample frames from the three major UGC datasets (LIVE-VQC, KoNViD-1k, and YouTube-UGC) demonstrating the wide variety of content and authentic distortions.

Key Insights and Analysis

Our benchmarking revealed several critical insights into the nature of UGC quality:

  • UGC is different: Traditional models designed for synthetic distortions fail to capture the complex mix of capture conditions, editing, and compression found in UGC.
  • Content Space Coverage: Different datasets cover different parts of the visual attribute space. As shown in the convex hull analysis, YouTube-UGC offers a much wider variety of spatial and temporal characteristics than mobile-only datasets like LIVE-VQC.
  • Resolution and Motion: Video quality is intrinsically correlated with resolution, and Natural Scene Statistics (NSS) features are implicitly "resolution-aware." Furthermore, for videos with heavy camera movement, motion-related features are crucial for accurate prediction.
Figure 2: Feature space coverage (convex hulls) for the three datasets across different visual attributes: Brightness (BR), Contrast (CT), Colorfulness (CF), Sharpness (SR), Spatial Information (SI), and Temporal Information (TI).

Experimental Results

VIDEVAL achieves state-of-the-art or near-SOTA results across all three individual datasets and the combined benchmark. Crucially, it is significantly faster than other high-performing models like FRIQUEE, making it more practical for real-world deployment. We also found that combining VIDEVAL with deep learning features (e.g., from KonCept512) can boost performance by another ~4%, pushing the upper bound of current blind VQA capabilities.

Conclusion and Future Work

While VIDEVAL performs well on natural videos, non-natural content (animations, gaming, screen recordings) remains a major challenge due to its underrepresentation in current datasets. Future research should focus on developing models for non-natural UGC, improving the handling of temporal distortions like frame drops, and enhancing cross-dataset generalization. The implementation of VIDEVAL has been made available online to facilitate reproducible research in this growing field.

Summary updated

Abstract

Recent years have witnessed an explosion of user-generated content (UGC) videos shared and streamed over the Internet, thanks to the evolution of affordable and reliable consumer capture devices, and the tremendous popularity of social media platforms. Accordingly, there is a great need for accurate video quality assessment (VQA) models for UGC/consumer videos to monitor, control, and optimize this vast content. Blind quality prediction of in-the-wild videos is quite challenging, since the quality degradations of UGC videos are unpredictable, complicated, and often commingled. Here we contribute to advancing the UGC-VQA problem by conducting a comprehensive evaluation of leading no-reference/blind VQA (BVQA) features and models on a fixed evaluation architecture, yielding new empirical insights on both subjective video quality studies and objective VQA model design. By employing a feature selection strategy on top of efficient BVQA models, we are able to extract 60 out of 763 statistical features used in existing methods to create a new fusion-based model, which we dub the VIDeo quality EVALuator (VIDEVAL), that effectively balances the trade-off between VQA performance and efficiency. Our experimental results show that VIDEVAL achieves state-of-the-art performance at considerably lower computational cost than other leading models. Our study protocol also defines a reliable benchmark for the UGC-VQA problem, which we believe will facilitate further research on deep learning-based VQA modeling, as well as perceptually-optimized efficient UGC video processing, transcoding, and streaming. To promote reproducible research and public evaluation, an implementation of VIDEVAL has been made available online: https://github.com/vztu/VIDEVAL.
BibTeX
@article{tu2020ugc,
  author = {Zhengzhong Tu and Yilin Wang and Neil Birkbeck and Balu Adsumilli and Alan C. Bovik},
  title = {{UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content}},
  journal = {IEEE Trans. Image Process.},
  year = {2020},
  doi = {10.1109/tip.2021.3072221},
}
Figure