On Evaluating Perceptual Quality of Online User-Generated Videos

Soobeom Jang, Jong-Seok Lee

Abstract

This paper deals with the issue of the perceptual quality evaluation of user-generated videos shared online, which is an important step toward designing video-sharing services that maximize users' satisfaction in terms of quality. We first analyze viewers' quality perception patterns by applying graph analysis techniques to subjective rating data. We then examine the performance of existing state-of-the-art objective metrics for the quality estimation of user-generated videos. In addition, we investigate the feasibility of metadata accompanied with videos in online video-sharing services for quality estimation. Finally, various issues in the quality assessment of online user-generated videos are discussed, including difficulties and opportunities.

Introduction

The explosion of user-generated videos (UGVs) on platforms like YouTube and Vimeo has created a "Wild West" of visual content. Unlike professional broadcasts, UGVs are characterized by diverse and often overlapping distortions—shaky cameras, poor lighting, and amateur editing—that occur before the video is even uploaded. Traditional No-Reference (NR) quality metrics, which were designed to detect compression artifacts in professional content, often fail to accurately assess the perceptual quality of these "in-the-wild" videos.

Subjective Analysis and Human Perception

To understand how humans perceive UGV quality, we conducted a large-scale subjective study using a paired comparison method. Over 8,400 comparisons were gathered for 50 YouTube videos. We applied HodgeRank analysis to decompose these choices into global quality scores and inconsistency measures.

HodgeRank analysis of subjective rating data, showing (a) pairwise comparison matrix, (b) estimated global scores, (c) residual inconsistencies, and (d) harmonic components.

A key finding was the high "total inconsistency" ratio of 67%. This suggests that human perception is highly subjective when videos exhibit different types of flaws (e.g., blurriness vs. shakiness), making it difficult for viewers to reach a consensus on which video is "better."

Benchmarking Objective Metrics

We evaluated three prominent NR metrics: V-BLIINDS, BRISQUE, and BLIINDS-II. These algorithms typically rely on natural scene statistics to identify pixel-level irregularities. The results were underwhelming; the best-performing metric, BRISQUE, achieved a Spearman Rank-Order Correlation Coefficient (SROCC) of only 0.33. These metrics often fail because they cannot distinguish between intentional aesthetic choices, such as high-motion time-lapses, and technical degradations.

The Power of Metadata

Given the limitations of signal-based analysis, we explored the use of video metadata (e.g., view counts, likes, description length) for quality estimation. We applied graph analysis techniques to the subjective data to further understand the structure of viewer preferences.

Modularity analysis of the subjective rating graph, illustrating the clustering of viewer perception patterns.

Our analysis revealed that metadata is a far more powerful predictor of quality than visual metrics. For instance, description length emerged as a strong proxy for uploader effort, correlating significantly with quality (SROCC 0.54). By combining 20 metadata features using Support Vector Regression (SVR), we achieved a remarkably high SROCC of 0.85. This demonstrates that for UGVs, the context surrounding the video is often more telling than the pixels themselves.

Conclusion and Future Work

This study highlights that traditional signal-processing metrics are insufficient for the complex landscape of online UGVs. Metadata-driven models offer a promising and computationally efficient alternative. Future research should focus on personalized quality assessment and developing metrics that can better interpret uploader intent versus technical failure.

Summary updated

Abstract

This paper deals with the issue of the perceptual quality evaluation of user-generated videos shared online, which is an important step toward designing video-sharing services that maximize users' satisfaction in terms of quality. We first analyze viewers' quality perception patterns by applying graph analysis techniques to subjective rating data. We then examine the performance of existing state-of-the-art objective metrics for the quality estimation of user-generated videos. In addition, we investigate the feasibility of metadata accompanied with videos in online video-sharing services for quality estimation. Finally, various issues in the quality assessment of online user-generated videos are discussed, including difficulties and opportunities.
BibTeX
@article{jang2016evaluating,
  author = {Soobeom Jang and Jong-Seok Lee},
  title = {{On Evaluating Perceptual Quality of Online User-Generated Videos}},
  journal = {IEEE Trans. Multimedia},
  year = {2016},
  url = {https://openreview.net/forum?id=RwfMAs61vW3},
}
Figure