DeepQoE: A unified Framework for Learning to Predict Video QoE

Huaizheng Zhang, Han Hu, Guanyu Gao, Yonggang Wen, Kyle Guan

Abstract

Predicting Video Quality of Experience (QoE) is a complex challenge due to the multi-faceted nature of human perception. Traditional objective models often rely on hand-crafted features that fail to generalize across different datasets or modalities. This paper introduces DeepQoE, a unified deep learning framework that automatically extracts features from multi-modal data (text, video, and metadata) to predict QoE. By employing a modular three-phase pipeline—preprocessing, representation learning, and task-specific prediction—DeepQoE handles both classification and regression tasks within a single architecture. Experimental results on large-scale datasets demonstrate that DeepQoE achieves a state-of-the-art accuracy of 90.94%, significantly outperforming traditional machine learning baselines in both performance and training efficiency.

Introduction

Measuring how a user perceives video quality (QoE) is significantly more complex than monitoring technical network statistics (QoS). QoE is influenced by a diverse set of "Influence Factors" (IFs), including system parameters (resolution, bitrate), contextual factors (device type), and human factors (user demographics).

Historically, QoE prediction has relied on expensive subjective tests or objective models limited by "hand-crafted" features. These traditional models often struggle to generalize to new content types and require distinct architectures for classification versus regression. DeepQoE addresses these limitations by providing an end-to-end pipeline that generalizes across different datasets and data modalities.

Methodology

The DeepQoE framework is structured into a three-phase pipeline designed to handle heterogeneous data sources and map them into a unified representation.

The DeepQoE framework architecture, illustrating the multi-modal preprocessing, representation learning, and task-specific prediction layers.

1. Feature Preprocessing

To manage data heterogeneity, specialized deep learning techniques map different data types into a shared high-dimensional feature space:

  • Textual Data: Movie descriptions or news categories are processed via pre-trained GloVe word embeddings.
  • Video Content: Raw video frames are processed using C3D (3D Convolutional Neural Networks) to capture both spatial information and temporal correlations.
  • Categorical Information: Metadata such as resolution or user gender is handled via an Embedding Layer.
  • Continuous Values: Parameters like bitrate or user age are normalized and passed through a Linear Layer.

2. Representation Learning

The extracted feature vectors are concatenated into a single unified vector. This vector is fed into a Deep Neural Network (DNN) consisting of multiple fully connected layers. The use of Dropout layers helps prevent overfitting, allowing the model to learn a robust unified representation of the video experience.

3. QoE Prediction

The final layer of the network is task-specific. For classification tasks (e.g., categorizing quality levels), a Softmax function is utilized. For regression tasks (e.g., predicting a specific Mean Opinion Score), a Linear function is used. This modularity allows the same core feature extractor to be repurposed for different analytical goals.

Experimental Evaluation

The framework was evaluated on two distinct benchmarks:

  • WHU-MVQoE2016: A small-scale dataset containing 72 video clips across four genres (movie, cartoon, news, sports) with 1,116 individual ratings.
  • Videoset: A large-scale dataset featuring 220 video titles and 45,760 clips. This dataset uses Just Noticeable Difference (JND) points as the ground truth, measuring the threshold of human perception for quality changes.

Results and Discussion

DeepQoE demonstrates significant improvements over traditional methods in accuracy, generalization, and computational efficiency.

Accuracy comparison on the large-scale Videoset dataset, where DeepQoE achieves 90.94% accuracy.

Performance at Scale

On the large-scale Videoset, DeepQoE achieved 90.94% accuracy, a substantial lead over the best traditional baseline, SVM, which reached 82.84%. Furthermore, the training process for DeepQoE was nearly 7x faster than SVM on the large dataset (504 seconds vs. 3,344 seconds), highlighting its scalability for modern big-data applications.

The Power of Learned Representations

A key insight from the study is the effectiveness of the learned features. Even "shallow" machine learning models, such as Random Forests or SVMs, showed improved performance when using the features extracted by the DeepQoE preprocessing layers compared to using raw features. This suggests that the framework successfully captures the underlying factors influencing QoE.

Generalization and Robustness

The model proved capable of predicting QoE for video types not seen during training (e.g., testing on movies after training on sports/news). On smaller datasets, DeepQoE remained competitive with shallow algorithms, proving that its learned representations are robust even when data is relatively scarce.

Conclusion and Future Work

DeepQoE provides a flexible and powerful framework for unified QoE prediction. While it excels at classification and large-scale analysis, the authors noted that regression accuracy on small datasets remains a challenge, as deep learning typically requires more data for fine-grained regression than simple linear models. Future work will focus on adapting the framework for real-time prediction in live streaming environments and incorporating attention mechanisms to identify specific video segments that most significantly impact user perception.

Summary updated

Abstract

Motivated by the prowess of deep learning (DL) based techniques in prediction, generalization, and representation learning, we develop a novel framework called DeepQoE to predict video quality of experience (QoE). The end-to-end framework first uses a combination of DL techniques (e.g., word embeddings) to extract generalized features. Next, these features are combined and fed into a neural network for representation learning. Such representations serve as inputs for classification or regression tasks. Evaluating the performance of DeepQoE with two datasets, we show that for the small dataset, the accuracy of all shallow learning algorithms is improved by using the representation derived from DeepQoE. For the large dataset, our DeepQoE framework achieves significant performance improvement in comparison to the best baseline method (90.94% vs. 82.84%). Moreover, DeepQoE, also released as an open source tool, provides video QoE research much-needed flexibility in fitting different datasets, extracting generalized features, and learning representations.
BibTeX
@inproceedings{zhang2018deepqoe,
  author = {Huaizheng Zhang and Han Hu and Guanyu Gao and Yonggang Wen and Kyle Guan},
  title = {{DeepQoE: A unified Framework for Learning to Predict Video QoE}},
  booktitle = {ICME},
  year = {2018},
  doi = {10.1109/icme.2018.8486523},
}
Figure