Abstract
Measuring Quality of Experience (QoE) and integrating these measurements into video streaming algorithms is a multi-faceted problem that fundamentally requires the design of comprehensive subjective QoE databases and metrics. To achieve this goal, we have recently designed the LIVE-NFLX-II database, a highly-realistic database which contains subjective QoE responses to various design dimensions, such as bitrate adaptation algorithms, network conditions and video content. Our database builds on recent advancements in content-adaptive encoding and incorporates actual network traces to capture realistic network variations on the client device. Using our database, we study the effects of multiple streaming dimensions on user experience and evaluate video quality and quality of experience models. We believe that the tools introduced here will help inspire further progress on the development of perceptually-optimized client adaptation and video streaming strategies.
Introduction
Modern video streaming relies on HTTP-based Adaptive Streaming (HAS), where players dynamically adjust video quality based on fluctuating network conditions. However, traditional optimization goals often focus on technical metrics like average bitrate, which do not always align with human perception. Viewers are sensitive to quality fluctuations, start-up delays, and rebuffering events. To bridge this gap, researchers from UT Austin and Netflix developed the LIVE-NFLX-II database, a large-scale resource designed to model human Quality of Experience (QoE) in realistic streaming scenarios.
The Streaming Pipeline and Methodology
The study employs an end-to-end pipeline to simulate the modern streaming experience, integrating content-adaptive encoding, realistic network traces, and various Adaptive Bitrate (ABR) algorithms.
The methodology consists of four key components:
- Content-Adaptive Encoding: Utilizing Netflix’s "Dynamic Optimizer," videos were encoded on a per-shot basis guided by VMAF (Video Multi-method Assessment Fusion) to ensure perceptual consistency.
- Network Simulation: The researchers used seven real-world 3G network traces (captured from mobile environments like trains and cars) to provide realistic bandwidth volatility.
- ABR Algorithms: Four strategies were evaluated: Buffer-Based (BB), Rate-Based (RB), Quality-Based (QB), and an "Oracle" Quality-Based (OQB) version with perfect future network knowledge.
- Subjective Testing: 65 viewers provided over 9,700 scores across 420 test videos, capturing both real-time (continuous) and overall (retrospective) quality assessments.
The LIVE-NFLX-II Database
The primary contribution of this work is the LIVE-NFLX-II database. It encompasses 15 unique video contents, ranging from high-action sports to cinematic scenes, ensuring a broad representation of visual complexity.
By combining these contents with diverse network traces and ABR strategies, the database provides a robust foundation for training and evaluating QoE models. It is publicly available to the research community to foster further innovation in perceptual streaming.
Key Insights and Results
The study revealed several critical aspects of human video perception:
- Psychological Effects: Viewers exhibit a Recency Effect, where the quality at the end of a video disproportionately affects the final score. Additionally, Duration Neglect was observed during rebuffering; once a pause exceeds approximately 4 seconds, the negative impact on QoE tends to plateau.
- Bitrate vs. Quality: Higher bitrates do not always equate to better experiences. Rate-based algorithms often proved too "greedy," leading to frequent rebuffering. In contrast, Quality-based (QB) approaches offered a superior balance between visual clarity and playback stability.
- VMAF as a Predictor: VMAF emerged as the most significant feature for predicting human QoE, contributing to 79% of the feature importance in the models tested.
- Model Performance: By training the Video ATLAS model on this new data, the researchers achieved a Spearman correlation (SROCC) of 0.87, significantly outperforming traditional metrics like PSNR (0.71).
Conclusion and Future Work
While LIVE-NFLX-II represents a significant step forward, limitations remain. The test sessions were relatively short (under 40 seconds), and factors like TCP slow-start and audio quality were not fully modeled. Future research aims to "close the loop" by integrating these perceptual models directly into ABR algorithms, enabling players to make real-time decisions that maximize human satisfaction rather than just technical throughput.
The database is available at: http://live.ece.utexas.edu/research/LIVE_NFLX_II/live_nflx_plus.html
Summary updated