Abstract
Predicting Video Quality of Experience (QoE) is a complex challenge due to the multi-faceted nature of human perception. Traditional objective models often rely on hand-crafted features that fail to generalize across different datasets or modalities. This paper introduces DeepQoE, a unified deep learning framework that automatically extracts features from multi-modal data (text, video, and metadata) to predict QoE. By employing a modular three-phase pipeline—preprocessing, representation learning, and task-specific prediction—DeepQoE handles both classification and regression tasks within a single architecture. Experimental results on large-scale datasets demonstrate that DeepQoE achieves a state-of-the-art accuracy of 90.94%, significantly outperforming traditional machine learning baselines in both performance and training efficiency.
Introduction
Measuring how a user perceives video quality (QoE) is significantly more complex than monitoring technical network statistics (QoS). QoE is influenced by a diverse set of "Influence Factors" (IFs), including system parameters (resolution, bitrate), contextual factors (device type), and human factors (user demographics).
Historically, QoE prediction has relied on expensive subjective tests or objective models limited by "hand-crafted" features. These traditional models often struggle to generalize to new content types and require distinct architectures for classification versus regression. DeepQoE addresses these limitations by providing an end-to-end pipeline that generalizes across different datasets and data modalities.
Methodology
The DeepQoE framework is structured into a three-phase pipeline designed to handle heterogeneous data sources and map them into a unified representation.
1. Feature Preprocessing
To manage data heterogeneity, specialized deep learning techniques map different data types into a shared high-dimensional feature space:
- Textual Data: Movie descriptions or news categories are processed via pre-trained GloVe word embeddings.
- Video Content: Raw video frames are processed using C3D (3D Convolutional Neural Networks) to capture both spatial information and temporal correlations.
- Categorical Information: Metadata such as resolution or user gender is handled via an Embedding Layer.
- Continuous Values: Parameters like bitrate or user age are normalized and passed through a Linear Layer.
2. Representation Learning
The extracted feature vectors are concatenated into a single unified vector. This vector is fed into a Deep Neural Network (DNN) consisting of multiple fully connected layers. The use of Dropout layers helps prevent overfitting, allowing the model to learn a robust unified representation of the video experience.
3. QoE Prediction
The final layer of the network is task-specific. For classification tasks (e.g., categorizing quality levels), a Softmax function is utilized. For regression tasks (e.g., predicting a specific Mean Opinion Score), a Linear function is used. This modularity allows the same core feature extractor to be repurposed for different analytical goals.
Experimental Evaluation
The framework was evaluated on two distinct benchmarks:
- WHU-MVQoE2016: A small-scale dataset containing 72 video clips across four genres (movie, cartoon, news, sports) with 1,116 individual ratings.
- Videoset: A large-scale dataset featuring 220 video titles and 45,760 clips. This dataset uses Just Noticeable Difference (JND) points as the ground truth, measuring the threshold of human perception for quality changes.
Results and Discussion
DeepQoE demonstrates significant improvements over traditional methods in accuracy, generalization, and computational efficiency.
Performance at Scale
On the large-scale Videoset, DeepQoE achieved 90.94% accuracy, a substantial lead over the best traditional baseline, SVM, which reached 82.84%. Furthermore, the training process for DeepQoE was nearly 7x faster than SVM on the large dataset (504 seconds vs. 3,344 seconds), highlighting its scalability for modern big-data applications.
The Power of Learned Representations
A key insight from the study is the effectiveness of the learned features. Even "shallow" machine learning models, such as Random Forests or SVMs, showed improved performance when using the features extracted by the DeepQoE preprocessing layers compared to using raw features. This suggests that the framework successfully captures the underlying factors influencing QoE.
Generalization and Robustness
The model proved capable of predicting QoE for video types not seen during training (e.g., testing on movies after training on sports/news). On smaller datasets, DeepQoE remained competitive with shallow algorithms, proving that its learned representations are robust even when data is relatively scarce.
Conclusion and Future Work
DeepQoE provides a flexible and powerful framework for unified QoE prediction. While it excels at classification and large-scale analysis, the authors noted that regression accuracy on small datasets remains a challenge, as deep learning typically requires more data for fine-grained regression than simple linear models. Future work will focus on adapting the framework for real-time prediction in live streaming environments and incorporating attention mechanisms to identify specific video segments that most significantly impact user perception.
Summary updated