TL;DR
This paper tackles the challenge of predicting the real-time subjective quality of HTTP video streams, which is crucial for optimizing streaming performance. The authors propose an "augmented" approach that combines multiple video quality assessment (VQA) models and uses a simple averaging ensemble of forecasts to significantly improve prediction accuracy over existing methods.
What Problem Does This Paper Address?
Online video streaming services like Netflix and YouTube use adaptive bitrate (ABR) strategies to adjust video quality based on network conditions, aiming to maximize user Quality of Experience (QoE). However, accurately predicting this subjective QoE in real-time is difficult due to the complex, non-linear nature of human perception and the dynamic changes in video quality caused by ABR. Existing methods often rely on a single input (like a single VQA metric) or a single prediction strategy, which can be insufficient. This paper addresses the need for a more robust and accurate continuous-time QoE prediction system that can better handle the complexities of modern video streaming.
Methods
The core of the proposed method is an augmented Non-linear Autoregressive (NARX) network. NARX models are well-suited for time-series prediction, capturing dependencies between past outputs and current inputs. The authors enhance the standard NARX approach in two key ways:
- Multiple VQA Inputs: Instead of relying on a single VQA metric (e.g., PSNR, SSIM), they feed multiple VQA scores from different models (e.g., SSIM, GMSD, ST-RRED) into the NARX network. The idea is that different VQA models capture different aspects of video distortion, and combining them provides a more comprehensive picture of video quality.
- Averaging Forecast Ensemble: To further improve robustness, they combine predictions from multiple NARX models. These models are trained with different configurations (e.g., varying input lags, different VQA input combinations). A simple averaging of these individual forecasts helps to filter out spurious predictions and reduce overall error.
They also compare two NARX training configurations: open-loop (OL), where ground truth QoE is used during training, and closed-loop (CL), where predicted QoE is fed back. While CL is necessary for testing, OL is found to be more computationally efficient and often yields better results for training.
Datasets
The primary dataset used for training and evaluation is the LIVE QoE Database for HTTP-based Video Streaming [11]. This database contains 15 video sequences, each 5 minutes long, encoded at various bitrates to simulate HTTP-based streaming with rate changes. The paper specifically notes that they did not use the LIVE-NFLX Video QoE Database [21] because their focus is on rate changes, not on combinations of rebuffering and dynamic rate drops.
Key Insights and Ideas
- Diversity is Key: Using multiple, diverse VQA models as inputs to the predictor is more effective than relying on a single, even if highly accurate, VQA metric. Different models capture different perceptual aspects, and their collective input provides a richer signal for QoE prediction.
- Ensembling for Robustness: Simple averaging of forecasts from multiple predictor configurations significantly improves prediction stability and accuracy, reducing the impact of individual model weaknesses or suboptimal configurations.
- Open-Loop Advantage: For training, the open-loop configuration of NARX is generally more efficient and can yield competitive or better results compared to the closed-loop configuration, especially when combined with ensembles.
- Complexity vs. Performance Trade-off: While more complex VQA models might seem appealing, the paper suggests that combining simpler, complementary VQA metrics can be highly effective and more practical for real-time applications.
Key Results
The augmented NARX approach, combining multiple VQA inputs and an averaging ensemble, demonstrated significant improvements in QoE prediction accuracy.
- Improved Metrics: The paper reports substantial gains in metrics like Outage Rate (OR), Spearman's Rank Order Correlation Coefficient (SROCC), and Pearson's Linear Correlation Coefficient (PLCC) compared to using single VQA inputs or simpler prediction methods. For instance, combining PSNR with NIQE showed large improvements, and further combining ST-RRED, GMSD, and SSIM yielded the best results.
- Best Combination: The best performing configuration identified was an open-loop NARX model using an averaging ensemble with three VQA inputs: ST-RRED, GMSD, and SSIM.
- Computational Efficiency: The OL configuration was found to be significantly less time-consuming for training than the CL configuration, with training times increasing with the number of VQA inputs.
Limitations and Future Work
- Limitations:
- The paper focuses solely on compression artifacts and rate changes, not explicitly addressing rebuffering events, which are another major impairment in video streaming.
- The ensemble method used is a simple averaging; more advanced ensemble techniques could potentially yield even better results.
- The study was conducted on a specific dataset; broader evaluation on other QoE databases is needed.
- Future Work:
- Investigate similar augmented approaches for dynamic systems like the Hammerstein-Wiener (HW) model.
- Extend the approach to predict QoE for videos affected by playback interruptions (rebuffering).
- Explore more sophisticated ensemble forecasting methods.
- Conduct extensive evaluations on other subjective QoE databases.
Summary updated