ML Models for Detecting QoE Degradation in Low-Latency Applications: A Cloud-Gaming Case Study

Joël Roman Ky, Bertrand Mathieu, Abdelkader Lahmadi, Raouf BoutabaIEEE Trans. Netw. Serv. Manag 2023 · DOI

Introduction

As the digital landscape shifts toward the "Metaverse" and ultra-low-latency applications—such as remote surgery and high-fidelity cloud gaming—Internet Service Providers (ISPs) face an unprecedented monitoring challenge. Ensuring a high Quality of Experience (QoE) requires detecting network degradations in near real-time. While unsupervised Machine Learning (ML) offers a promising path forward, the current research landscape is fragmented by inconsistent methodologies and evaluation metrics that often fail to reflect real-world operational needs.

This study benchmarks eight unsupervised ML models for detecting QoE degradation in cloud gaming, introducing a novel evaluation framework designed to align model performance with human perception.

The Limitations of Point-Wise Detection

Traditional anomaly detection research often relies on "point-wise" evaluation, where a model is tasked with flagging individual data points (e.g., a single 5ms sample). However, this approach is fundamentally misaligned with human experience. A 5ms glitch is imperceptible to a gamer; humans typically perceive degradation over longer windows of time, generally exceeding 100ms.

Furthermore, common metrics like the F1-score are prone to overestimating a model's effectiveness. For instance, a "zero-rule" model that flags every point as an anomaly can achieve a misleadingly high F1-score of approximately 70%. To address this, the authors advocate for the use of the Matthews Correlation Coefficient (MCC), which correctly identifies such trivial models as ineffective by assigning them a score of zero.

The Window Anomaly Decision (WAD) Framework

The core innovation of this research is the Window Anomaly Decision (WAD) approach. Unlike the popular "Point-Adjust" (PA) method—which grants a model full credit for an entire anomalous segment if it detects just a single point—WAD requires a specific density of anomalies (e.g., 80%) within a defined time window to trigger a detection.

Comparison of anomaly detection approaches: WAD vs. traditional point-wise and point-adjust methods.

As shown in the comparison above, WAD prevents "lucky guesses" from inflating performance scores. By requiring sustained evidence of degradation, it ensures that detected events are long enough to actually impact a user's experience, providing a more reliable metric for ISPs.

Experimental Methodology

The authors evaluated eight unsupervised models across three categories:

  1. Isolation-based: iForest (Isolation Forest).
  2. One-class classification: OC-SVM and Deep-SVDD.
  3. Reconstruction-based: PCA, AutoEncoder (AE), LSTM-VAE, DAGMM, and USAD.

These models were tested using real-world multivariate time-series data from three major platforms: Google Stadia (STD), Nvidia GeForce Now (GFN), and Microsoft Xbox Cloud (XC). The dataset included 14 features, such as Round Trip Time (RTT), jitter buffer delay, and frame rate, collected under various 4G network conditions. To test robustness, the authors introduced "contamination sets," intentionally adding anomalous samples into the training data to simulate noisy, real-world environments.

Key Results and Discussion

  • Top Performers: USAD and DAGMM emerged as the most reliable models, maintaining high MCC scores (~82-84% on the Stadia dataset) even under heavy training data contamination.
  • The "Clean Data" Myth: While deep learning models like Deep-SVDD and LSTM-VAE perform near-optimally on perfectly clean data, their performance plummets when training data is contaminated with as little as 4% anomalies.
  • iForest’s Robustness: Interestingly, Isolation Forest (iForest) performed better when training data contained moderate noise, as the noise helped the model better define the boundaries of "normal" behavior.
  • Efficiency: Most deep learning models (USAD, DAGMM, AE) achieved inference times between 89µs and 126µs per window, making them highly suitable for real-time monitoring. In contrast, OC-SVM proved impractical for ISP use cases, taking over 12 hours to train.

Conclusion

Standardizing the evaluation of QoE detection models is critical for the deployment of ML in network monitoring. The WAD approach provides a more perceptually accurate benchmark than traditional methods. Future work will focus on moving from client-side API features to network-edge packet analysis, allowing ISPs to detect issues directly within the infrastructure without requiring access to user devices or internal application metrics.

Summary updated

BibTeX
@article{ky2023ml,
  author = {Joël Roman Ky and Bertrand Mathieu and Abdelkader Lahmadi and Raouf Boutaba},
  title = {{ML Models for Detecting QoE Degradation in Low-Latency Applications: A Cloud-Gaming Case Study}},
  journal = {IEEE Trans. Netw. Serv. Manag.},
  year = {2023},
  doi = {10.1109/tnsm.2023.3293806},
}
Figure