Abstract
The proliferation of mobile videography has introduced a vast array of authentic video impairments that traditional no-reference (NR) video quality assessment (VQA) models struggle to quantify. This paper introduces the LIVE Video Quality Challenge (LIVE-VQC) Database, a large-scale resource containing 585 unique, user-generated videos with over 205,000 subjective human opinion scores. Our study demonstrates that while current algorithms excel on synthetic lab-based distortions, they falter when faced with the complex, commingled artifacts of real-world content.
Introduction: The Challenge of Authentic Distortions
Most existing VQA models are developed using "pristine" videos degraded by controlled, synthetic distortions like H.264 compression or white noise. However, real-world user-generated content (UGC) on platforms like YouTube or Instagram is "authentically" distorted. These videos often suffer from a chaotic mixture of camera shake, poor lighting, autofocus hunting, and sensor noise—often occurring simultaneously.
Current NR models—which must estimate quality without a reference video—frequently fail in these "in the wild" scenarios. This research aims to bridge the gap by providing a massive, diverse dataset reflecting the true state of modern mobile videography.
Methodology: Crowdsourcing at Scale
To collect high-quality subjective scores, the researchers utilized Amazon Mechanical Turk (AMT) with several rigorous controls to ensure "science-quality" data:
- Hardware Requirements: Participants were restricted to non-mobile devices with at least 720p resolution.
- Playback Integrity: Videos were forced to fully preload to prevent network stalls from biasing scores. If a hardware-related stall occurred, it was logged and the data filtered.
- Quality Control: "Golden Videos" with known quality scores from previous lab studies were interspersed to flag and filter unreliable participants.
- Subjective Scaling: Ratings were collected on a continuous Likert-like scale (Bad to Excellent) and converted into Mean Opinion Scores (MOS).
The LIVE-VQC Dataset
The primary contribution is the LIVE-VQC Database, featuring 585 unique 10-second videos captured by 80 different users across 101 mobile devices. The dataset captures a wide variety of scenes, from low-light concerts to high-detail landscapes, representing the vast diversity of real-world capture conditions.
Analysis of the 205,000 collected scores revealed several key insights. First, MOS values stabilize after approximately 200 ratings per video, providing a heuristic for future studies. Second, a generational bias was observed: younger participants (under 20) were consistently harsher critics than older participants. Finally, human perception of quality proved robust across different viewing distances and screen resolutions, provided the resolution was above 720p.
Results and Algorithm Benchmarking
The study evaluated leading NR-VQA algorithms, including NIQE, VIIDEO, BRISQUE, and V-BLIINDS. While these models often achieve Spearman Rank Ordered Correlation (SROCC) scores above 0.90 on synthetic datasets, their performance plummeted on LIVE-VQC. V-BLIINDS was the top performer with an SROCC of 0.7083, whereas VIIDEO failed to correlate with human judgment entirely (SROCC -0.0293). This highlights the significant "performance gap" between lab-based and real-world VQA.
Conclusion
The LIVE-VQC database provides a critical benchmark for the next generation of video quality models. By moving beyond synthetic distortions to authentic, "in the wild" content, researchers can develop more robust algorithms that better align with human perception in the diverse landscape of modern digital media. Future work will focus on developing "stall-aware" models and creating taxonomies for commingled real-world impairments.
Summary updated