Abstract
No-reference (NR) perceptual video quality assessment (VQA) is a complex, unsolved, and important problem for social and streaming media applications. Efficient and accurate video quality predictors are needed to monitor and guide the processing of billions of shared, often imperfect, user-generated content (UGC). Unfortunately, current NR models are limited in their prediction capabilities on real-world, "in-the-wild" UGC video data. To advance progress on this problem, we created the largest (by far) subjective video quality dataset, containing 38,811 real-world distorted videos and 116,433 space-time localized video patches ('v-patches'), and 5.5M human perceptual quality annotations. Using this, we created two unique NR-VQA models: (a) a local-to-global region-based NR VQA architecture (called PVQ) that learns to predict global video quality and achieves state-of-the-art performance on 3 UGC datasets, and (b) a first-of-a-kind space-time video quality mapping engine (called PVQ Mapper) that helps localize and visualize perceptual distortions in space and time.
Introduction: The UGC Quality Challenge
As user-generated content (UGC) explodes on platforms like YouTube and TikTok, measuring video quality has become a massive technical hurdle. Traditional "Full-Reference" metrics, which compare a distorted video to a pristine original, are inapplicable to UGC because a "perfect original" rarely exists.
"No-Reference" (NR) models are the necessary solution, but they face three major challenges:
- Data Scarcity: Existing datasets were historically too small to train modern, data-hungry deep networks.
- Complexity of Distortions: UGC suffers from a messy mix of "in-the-wild" artifacts—shaky hands, poor lighting, compression glitches, and frame drops—that are much harder to model than synthetic blur or noise.
- The Global-Local Gap: A video might look acceptable overall but contain a two-second glitch or a blurry corner. Most models struggle to relate these local "patches" of poor quality to a viewer's overall impression.
The LSVQ Dataset
To address the data gap, the authors introduced the Large-Scale Social Video Quality (LSVQ) Database. At 38,811 real-world distorted videos, it is 26x larger than previous UGC datasets. Crucially, it includes 116,433 space-time localized "v-patches," supported by 5.5 million subjective quality annotations from over 6,200 human subjects.
By collecting human scores for these tiny crops of video, the authors provided the model with "local" supervision. This teaches the network what a "bad patch" looks like, which it then uses to better understand the "global" video quality.
The Patch-VQ Architecture
The proposed Patch-VQ (PVQ) model processes video through two parallel streams to capture the full spectrum of perceptual quality:
- 2D Stream: Uses a PaQ-2-PiQ backbone (pre-trained on images) to capture spatial quality and frame content.
- 3D Stream: Uses a ResNet3D-18 backbone to capture temporal distortions like flicker, stutter, and motion blur.
These features are passed through RoIPool (Region-of-Interest Pooling) and SoIPool (Segment-of-Interest Pooling) layers, allowing the model to analyze specific spatial regions and temporal segments. Finally, an InceptionTime regressor—a state-of-the-art model for time-series analysis—predicts the final quality score by analyzing how quality fluctuates over the video's duration.
Additionally, the PVQ Mapper engine can generate "heat maps" of quality by applying the model to a grid of space-time patches, visualizing exactly where and when a video is failing.
Key Results and Insights
PVQ achieved state-of-the-art (SOTA) performance on the LSVQ dataset, with a Spearman Rank Correlation Coefficient (SRCC) of 0.827, significantly outperforming previous benchmarks.
- 3D Features are Critical: Ablation studies showed that the 3D (spatio-temporal) stream consistently outperformed the 2D stream, confirming that motion and temporal changes are vital for video quality perception.
- Superior Generalization: When trained on LSVQ and tested on other datasets (like LIVE-VQC) without fine-tuning, PVQ still outperformed models specifically designed for those datasets. This suggests the model has learned fundamental "quality" features rather than dataset-specific quirks.
- Temporal Modeling: The authors found that InceptionTime was more effective than traditional RNNs at capturing perceptual quality changes without suffering from vanishing gradients.
Conclusion and Future Work
While PVQ represents a major leap forward, challenges remain. The model occasionally struggles to distinguish between technical errors and intentional artistic choices, such as "bokeh" background blur. Future work may involve improving content diversity handling and optimizing the 3D ResNet-based architecture for real-time applications on mobile devices.
Summary updated