Taming the Latency in Multi-User VR 360°: A QoE-Aware Deep Learning-Aided Multicast Framework

Cristina Perfecto, Mohammed Saad ElBamby, Javier Del Ser, Mehdi Bennis

TL;DR

This paper introduces a novel framework for delivering high-quality, low-latency 360° VR video streams to multiple users simultaneously. By using deep learning to predict user viewpoints and employing a matching theory approach for resource allocation over mmWave networks, the system significantly reduces latency and improves the overall VR experience.

What Problem Does This Paper Address?

Immersive virtual reality (VR) experiences, especially 360° video streaming, demand extremely high data rates and ultra-low latency (motion-to-photon delay below 15-20 ms) to prevent VR sickness and ensure user engagement. Current wireless solutions struggle to meet these stringent requirements, particularly in multi-user scenarios where bandwidth and latency constraints become even more critical. Existing approaches often rely on reactive methods or don't fully leverage the spatial and temporal correlations between users' viewing patterns, leading to inefficient resource utilization and suboptimal quality of experience (QoE). This paper aims to bridge this gap by developing a proactive, QoE-aware system that intelligently predicts user needs and optimizes resource allocation in 5G mmWave networks.

Methods

The proposed framework combines several key techniques:

  • Deep Recurrent Neural Network (DRNN) for FoV Prediction: A DRNN, specifically using Gated Recurrent Units (GRUs), is employed to predict users' future Field of View (FoV) based on their past head movements. This proactive prediction allows the system to anticipate content needs.
  • FoV and Location-Aware User Clustering: Users with overlapping predicted FoVs and spatial proximity are grouped into clusters. This enables efficient multicast transmission of content.
  • Lyapunov Optimization Framework: This framework is used to manage the trade-off between maximizing HD frame delivery rates and maintaining stable traffic queues and strict latency constraints. It transforms the complex stochastic optimization problem into a series of manageable instantaneous subproblems.
  • Matching Theory for Resource Allocation: A matching theory-based algorithm is used to schedule the transmission of HD video chunks. It pairs user clusters with mmWave Small Cell Base Stations (SBSs) to optimize resource allocation, leveraging the directional capabilities of mmWave.
  • Proactive Multicast Transmission: By predicting FoVs and clustering users, the system can proactively schedule multicast transmissions of video chunks, reducing redundant data delivery and latency.

Datasets

The paper utilizes a real VR head-tracking dataset for training and simulation. This dataset consists of 3DoF pose data from 50 different users watching 10 different HD 360° videos. The dataset is split into training (35 users) and testing (15 users) sets.

Key Insights and Ideas

  • Predicting the Future is Key: Proactively predicting user FoVs is crucial for efficient multicast transmission and latency reduction in VR streaming. The DRNN's ability to anticipate user views is a core enabler.
  • Clustering for Efficiency: Grouping users with similar viewing patterns significantly enhances content reuse, a fundamental principle for multicast efficiency.
  • Matching Theory for Optimal Resource Allocation: Applying matching theory to pair user clusters with SBSs provides a robust and efficient mechanism for dynamic resource allocation in a complex, multi-user environment.
  • mmWave's Role: The directional nature of mmWave communication, combined with multi-beam capabilities, is essential for supporting the high bandwidth and targeted delivery required for VR.
  • Balancing Quality and Latency: The Lyapunov optimization framework provides a principled way to manage the inherent trade-off between delivering the highest possible video quality and strictly adhering to latency bounds.

Key Results

  • The proposed proactive multicast framework significantly reduces VR frame delay by 12% due to content reuse.
  • Proactiveness alone cuts average delays by half compared to reactive unicast and multicast baselines.
  • HD delivery rates are maintained above 98%.
  • Enforcing tight latency bounds shortens the delay-tail, resulting in 13% lower delays in the 99th percentile.
  • The DRNN achieves good prediction accuracy, with Jaccard similarity indices generally above 0.5, even for prediction horizons of up to 30 frames.
  • Multicasting provides a 40-50% increase in HD rate and 33-70% latency reduction compared to unicast.

Limitations and Future Work

  • Prediction Accuracy: While good, prediction accuracy decreases with longer prediction horizons, impacting the effectiveness of proactive scheduling.
  • Inter-SBS Interference: The paper acknowledges that the matching game cannot be directly solved using standard DA due to externalities (interference). While an interference estimation method is proposed, managing this dynamic interference remains a challenge.
  • Future Work: The authors suggest exploring more sophisticated utility functions for proportional fairness and investigating the impact of other edge computing aspects like caching on the overall system performance. Further research could also focus on adaptive prediction horizons and more advanced DRNN architectures.

Summary updated

Abstract

Immersive virtual reality (VR) applications require ultra-high data rate and low-latency for smooth operation. Hence in this paper, aiming to improve VR experience in multi-user VR wireless video streaming, a deep-learning aided scheme for maximizing the quality of the delivered video chunks with low-latency is proposed. Therein the correlations in the predicted field of view (FoV) and locations of viewers watching 360° HD VR videos are capitalized on to realize a proactive FoV-centric millimeter wave (mmWave) physical-layer multicast transmission. The problem is cast as a frame quality maximization problem subject to tight latency constraints and network stability. The problem is then decoupled into an HD frame request admission and scheduling subproblems and a matching theory game is formulated to solve the scheduling subproblem by associating requests from clusters of users to mmWave small cell base stations (SBSs) for their unicast/multicast transmission. Furthermore, for realistic modeling and simulation purposes, a real VR head-tracking dataset and a deep recurrent neural network (DRNN) based on gated recurrent units (GRUs) are leveraged. Extensive simulation results show how the content-reuse for clusters of users with highly overlapping FoVs brought in by multicasting reduces the VR frame delay in 12%. This reduction is further boosted by proactiveness that cuts by half the average delays of both reactive unicast and multicast baselines while preserving HD delivery rates above 98%. Finally, enforcing tight latency bounds shortens the delay-tail as evinced by 13% lower delays in the 99th percentile.
BibTeX
@article{perfecto2020taming,
  author = {Cristina Perfecto and Mohammed Saad ElBamby and Javier Del Ser and Mehdi Bennis},
  title = {{Taming the Latency in Multi-User VR 360°: A QoE-Aware Deep Learning-Aided Multicast Framework}},
  journal = {IEEE Trans. Communications},
  year = {2020},
  url = {https://openreview.net/forum?id=O0A9ztqTyS},
}
Figure