Large-Scale Study of Perceptual Video Quality

Zeina Sinno, Alan C. Bovik

Abstract

The proliferation of mobile videography has introduced a vast array of authentic video impairments that traditional no-reference (NR) video quality assessment (VQA) models struggle to quantify. This paper introduces the LIVE Video Quality Challenge (LIVE-VQC) Database, a large-scale resource containing 585 unique, user-generated videos with over 205,000 subjective human opinion scores. Our study demonstrates that while current algorithms excel on synthetic lab-based distortions, they falter when faced with the complex, commingled artifacts of real-world content.

Introduction: The Challenge of Authentic Distortions

Most existing VQA models are developed using "pristine" videos degraded by controlled, synthetic distortions like H.264 compression or white noise. However, real-world user-generated content (UGC) on platforms like YouTube or Instagram is "authentically" distorted. These videos often suffer from a chaotic mixture of camera shake, poor lighting, autofocus hunting, and sensor noise—often occurring simultaneously.

Current NR models—which must estimate quality without a reference video—frequently fail in these "in the wild" scenarios. This research aims to bridge the gap by providing a massive, diverse dataset reflecting the true state of modern mobile videography.

Methodology: Crowdsourcing at Scale

To collect high-quality subjective scores, the researchers utilized Amazon Mechanical Turk (AMT) with several rigorous controls to ensure "science-quality" data:

  • Hardware Requirements: Participants were restricted to non-mobile devices with at least 720p resolution.
  • Playback Integrity: Videos were forced to fully preload to prevent network stalls from biasing scores. If a hardware-related stall occurred, it was logged and the data filtered.
  • Quality Control: "Golden Videos" with known quality scores from previous lab studies were interspersed to flag and filter unreliable participants.
  • Subjective Scaling: Ratings were collected on a continuous Likert-like scale (Bad to Excellent) and converted into Mean Opinion Scores (MOS).

The LIVE-VQC Dataset

The primary contribution is the LIVE-VQC Database, featuring 585 unique 10-second videos captured by 80 different users across 101 mobile devices. The dataset captures a wide variety of scenes, from low-light concerts to high-detail landscapes, representing the vast diversity of real-world capture conditions.

Figure 3: An example of diverse, high-detail content from the LIVE-VQC database featuring international flags in a salt flat environment.
Figure 6: A concert scene illustrating complex lighting and motion artifacts typical of user-generated content in the dataset.

Analysis of the 205,000 collected scores revealed several key insights. First, MOS values stabilize after approximately 200 ratings per video, providing a heuristic for future studies. Second, a generational bias was observed: younger participants (under 20) were consistently harsher critics than older participants. Finally, human perception of quality proved robust across different viewing distances and screen resolutions, provided the resolution was above 720p.

Results and Algorithm Benchmarking

The study evaluated leading NR-VQA algorithms, including NIQE, VIIDEO, BRISQUE, and V-BLIINDS. While these models often achieve Spearman Rank Ordered Correlation (SROCC) scores above 0.90 on synthetic datasets, their performance plummeted on LIVE-VQC. V-BLIINDS was the top performer with an SROCC of 0.7083, whereas VIIDEO failed to correlate with human judgment entirely (SROCC -0.0293). This highlights the significant "performance gap" between lab-based and real-world VQA.

Conclusion

The LIVE-VQC database provides a critical benchmark for the next generation of video quality models. By moving beyond synthetic distortions to authentic, "in the wild" content, researchers can develop more robust algorithms that better align with human perception in the diverse landscape of modern digital media. Future work will focus on developing "stall-aware" models and creating taxonomies for commingled real-world impairments.

Summary updated

Abstract

The great variations of videographic skills in videography, camera designs, compression and processing protocols, communication and bandwidth environments, and displays leads to an enormous variety of video impairments. Current noreference (NR) video quality models are unable to handle this diversity of distortions. This is true in part because available video quality assessment databases contain very limited content, fixed resolutions, were captured using a small number of camera devices by a few videographers and have been subjected to a modest number of distortions. As such, these databases fail to adequately represent real world videos, which contain very different kinds of content obtained under highly diverse imaging conditions and are subject to authentic, complex and often commingled distortions that are difficult or impossible to simulate. As a result, NR video quality predictors tested on real-world video data often perform poorly. Towards advancing NR video quality prediction, we have constructed a largescale video quality assessment database containing 585 videos of unique content, captured by a large number of users, with wide ranges of levels of complex, authentic distortions. We collected a large number of subjective video quality scores via crowdsourcing. A total of 4776 unique participants took part in the study, yielding more than 205000 opinion scores, resulting in an average of 240 recorded human opinions per video. We demonstrate the value of the new resource, which we call the LIVE Video Quality Challenge Database (LIVE-VQC for short), by conducting a comparison of leading NR video quality predictors on it. This study is the largest video quality assessment study ever conducted along several key dimensions: number of unique contents, capture devices, distortion types and combinations of distortions, study participants, and recorded subjective scores. The database is available for download on this link: http://live.ece.utexas.edu/research/LIVEVQC/index.html.
BibTeX
@article{sinno2018large,
  author = {Zeina Sinno and Alan C. Bovik},
  title = {{Large-Scale Study of Perceptual Video Quality}},
  journal = {IEEE Transactions on Image Processing},
  year = {2018},
  doi = {10.1109/tip.2018.2869673},
}
Figure