TL;DR
This paper tackles the challenge of predicting Quality of Experience (QoE) for video streaming by leveraging machine learning on network Quality of Service (QoS) metrics. The authors propose a novel Bayesian Network model for predicting re-buffering events, outperforming existing methods, and demonstrate that incorporating hidden variables and context information significantly boosts prediction accuracy for various QoE factors.
What Problem Does This Paper Address?
The increasing dominance of video traffic necessitates a shift from traditional Quality of Service (QoS) metrics (network-centric, quantitative) to Quality of Experience (QoE) metrics (user-centric, subjective). While QoS metrics are readily available, directly measuring QoE is difficult. This is especially true for factors like re-buffering events and startup delay, which are not directly observable from network measurements, particularly with end-to-end encryption used by many Over-The-Top (OTT) services. This paper aims to bridge this gap by using machine learning to predict key QoE factors from observable QoS data, enabling network operators to optimize for user satisfaction.
Methods
The core of the paper focuses on predicting three key QoE factors: average video bitrate, video bitrate variation, and re-buffering ratio.
- Re-buffering Prediction: The authors propose a Bayesian Network (BN) model that utilizes two Logistic Regression (LR) classifiers. This BN approach is shown to provide a better balance in class accuracies compared to Random Forests, a common benchmark. They also explore a novel Neural Network (NN) search method to investigate the possibility of linearly separable data classes, ultimately concluding that their BN model is likely optimal for the given dataset.
- Hidden Variables: A significant contribution is the exploration of using "hidden variables" – metrics that are not directly observable but can be inferred from QoS data. The paper proposes a method where these hidden variables are predicted first and then used as features to predict the target QoE factors.
- Contextual Information: The study also investigates the impact of incorporating "context information" (e.g., network congestion, video stream characteristics) on prediction accuracy.
- Video Bitrate and Variation: For these factors, which are more directly related to observable QoS metrics, a classic Regression Tree (RT) model, specifically a bagged RT, is found to be highly effective.
Datasets
The authors created a high-fidelity and fully controllable simulation environment using the Adaptive Multimedia Streaming Simulator Framework (AMust) in ns-3. This environment implements an HTTP client and server for LibDASH.
- Content: Three representative open movies (Big Buck Bunny, Swiss Account, Red Bull Play Street) were used, varying in motion scenes.
- Scenarios: A wide range of scenarios were simulated, varying network parameters (number of nodes, bottleneck capacity, delay, packet loss), client screen resolutions, and DASH policies (Rate-Based, Buffer-Based, hybrid).
- Data Collected: The dataset comprises statistics from over 69,000 video sessions, including context information, QoS metrics, target QoE factors, and hidden QoE variables. The dataset is intended to be made public.
Key Insights and Ideas
- Re-buffering is Complex: Re-buffering events are not easily predictable from simple linear relationships with QoS metrics; they reside in high-dimensional clusters, making them a challenging prediction task.
- Bayesian Networks Excel for Stalling: The proposed Bayesian Network model, using cascaded Logistic Regressions, offers a more balanced prediction accuracy across different re-buffering classes (NoStall, MildStall, SevereStall) compared to Random Forests.
- Hidden Variables Boost Performance: Predicting and incorporating hidden variables as features significantly improves the accuracy of QoE factor predictions, even when these hidden variables are themselves predicted.
- Context is King (for QoE): Adding context information about network conditions and video stream characteristics further enhances prediction accuracy for all QoE factors.
- Simpler Models for Simpler Problems: For QoE factors like average video bitrate and its variation, which have a stronger linear correlation with QoS, simpler models like Regression Trees are highly effective.
Key Results
- StallLabel Prediction: The BN model achieved competitive accuracies, outperforming a Random Forest benchmark. For example, validation accuracies for NoStall, MildStall, and SevereStall were 0.8684, 0.8048, and 0.9368 respectively.
- Impact of Hidden Variables: Using predicted hidden variables improved prediction accuracy. When actual hidden variables were used (perfect prediction), accuracies were significantly higher than using QoS metrics alone. For instance, validation accuracy for SevereStall increased from 0.9267 (QoS only) to 0.9530 (with actual hidden variables).
- Impact of Context: Adding predicted context information also improved accuracy, though to a lesser extent than actual hidden variables.
- Video Bitrate and Variation: Bagged Regression Trees achieved high accuracy for predicting average video bitrate and its variation. For example, the prediction error for average video quality variation was as low as 26.23 (0.13) with hidden and actual context information.
Limitations and Future Work
- Linear Separability: While the BN model is effective, the authors explored if data classes were linearly separable, suggesting that a more complex switching model (like a Neural Network) could potentially yield higher accuracy. However, their NN search did not find areas of consistent misclassification that would strongly benefit from such a model for this specific dataset.
- Model Complexity for Hidden Variables: The paper notes that while their simple predictors for hidden variables help, higher gains could be achieved with better models for these hidden variables.
- Future Work: The authors plan to design new features specifically for video profiling, measurable by QoS monitoring systems, to further improve QoE predictions. They also suggest exploring more sophisticated models for predicting hidden variables.
Summary updated