Virtual Reality in Metaverse over Wireless Networks with User-centered Deep Reinforcement Learning

Wenhan Yu, Terence Jie Chua, Jun Zhao

Abstract

The Metaverse promises a new era of highly immersive and interactive socialization, with Virtual Reality (VR) serving as its technological backbone. However, the high-resolution and high-frame-rate requirements of hyper-realistic 3D scenes often exceed the processing capabilities of standalone VR headsets. To address this, computation offloading to edge servers via wireless networks is essential. This paper presents a novel approach to managing multi-user VR computation offloading using a user-centered deep reinforcement learning framework. By introducing Hybrid Reward Proximal Policy Optimization (HRPPO), we demonstrate a method that prioritizes individual user experiences within a shared network environment, achieving significant performance gains and faster convergence compared to traditional methods.

Introduction

The realization of a seamless Metaverse hinges on the ability to deliver consistent, high-fidelity VR experiences to multiple concurrent users. As VR scenes become increasingly complex, the computational burden grows, necessitating the offloading of rendering tasks to edge servers. While wireless communication enables this offloading, it introduces significant challenges in resource management, particularly when multiple users share limited bandwidth. Traditional optimization strategies often focus on aggregate network metrics, which can lead to uneven user experiences where some participants suffer from latency or reduced visual quality while others remain unaffected. This research proposes a shift toward a user-centered paradigm, ensuring that the "Quality of Experience" (QoE) for every individual is accounted for in the network's decision-making process.

Problem Statement: The Computation Gap in Wireless VR

To maintain immersion and prevent motion sickness, VR requires resolutions of at least 2K and frame rates of 90 FPS or higher. Most mobile VR headsets lack the local hardware to sustain these specifications for complex Metaverse environments. Computation offloading allows the headset to send tracking data to a Virtual Service Provider (VSP), which renders the frames and streams them back to the user.

In a multi-user scenario, this process typically utilizes Non-Orthogonal Multiple Access (NOMA) to maximize spectral efficiency. However, NOMA introduces intra-cluster interference, and the shared nature of the wireless channel means that one user's high-bandwidth demand can degrade the connection for others. Furthermore, users have heterogeneous needs; a user in a high-intensity virtual game requires lower latency than a user in a static social lounge. Managing these trade-offs requires an intelligent, adaptive resource management system.

Methodology: User-Centered HRPPO

The core contribution of this work is the development of HRPPO (Hybrid Reward Proximal Policy Optimization). This algorithm builds upon two foundational concepts in reinforcement learning:

  1. Proximal Policy Optimization (PPO): A robust reinforcement learning algorithm that ensures stable policy updates, preventing the system from making erratic changes that could disrupt the user experience.
  2. Hybrid Reward Architecture (HRA): Unlike standard RL, which uses a single global reward signal, HRA decomposes the reward. In HRPPO, the AI agent receives distinct feedback for every individual user.

In this architecture, the "Critic" network estimates the state-value for each user independently. This allows the "Actor" network—the component responsible for channel assignment and offloading decisions—to understand the specific impact of its actions on each participant. The system model allows for a hybrid approach: users can either offload tasks to the VSP for high-quality rendering or compute locally at a lower resolution to conserve network bandwidth, albeit at the cost of device battery life.

Experimental Setup

The researchers evaluated HRPPO using a dynamic numerical simulation designed to mimic a high-density indoor Metaverse environment:

  • Environment: A 30m x 30m indoor space.
  • User Density: 5 to 8 users sharing 3 wireless channels.
  • VR Specifications: 90 FPS target with resolutions ranging from 1080p to 2K.
  • Channel Model: Rayleigh fading was used to simulate realistic wireless signal fluctuations.
  • Training: The model was trained over 200,000 steps using global random seeds to ensure the reproducibility of the results.

Key Results and Discussion

The experimental results demonstrate that HRPPO significantly outperforms baseline algorithms, including standard PPO and Hybrid Reward Deep Q-Networks (HRDQN).

  • Performance Gains: In high-congestion scenarios (8 users), HRPPO achieved a final reward approximately 45% higher than standard PPO. This indicates a much more efficient allocation of resources that balances the needs of all users.
  • Convergence Speed: HRPPO reached peak performance in roughly half the training steps required by standard PPO in a 6-user test. This faster learning is attributed to the decomposed reward, which reduces the "noise" in the feedback signal, allowing the agent to identify optimal strategies more quickly.
  • Resource Efficiency: In the 6-user scenario, HRPPO achieved a 100% success rate in offloading tasks. This resulted in zero energy consumption on the local VR devices for rendering, preserving battery life without sacrificing frame rates.
  • Throughput: The system maintained a significantly higher average downlink transmission rate (approx. 28 Mbps for 5 users) compared to traditional resource management baselines.

Conclusion and Future Work

This research proves that a user-centered approach to reinforcement learning is highly effective for managing the complex demands of Metaverse VR over wireless networks. By treating individual user experiences as distinct rewards, HRPPO ensures a fairer and more efficient distribution of computational and communication resources.

Future iterations of this work will explore dynamic power allocation within the NOMA system to further enhance spectral efficiency. Additionally, the researchers aim to incorporate user mobility models and investigate the scalability of the HRPPO architecture as the number of concurrent users in the virtual environment continues to grow.

Summary updated

Abstract

The Metaverse and its promises are fast becoming reality as maturing technologies are empowering the different facets. One of the highlights of the Metaverse is that it offers the possibility for highly immersive and interactive socialization. Virtual reality (VR) technologies are the backbone for the virtual universe within the Metaverse as they enable a hyper-realistic and immersive experience, and especially so in the context of socialization. As the virtual world 3D scenes to be rendered are of high resolution and frame rate, these scenes will be offloaded to an edge server for computation. Besides, the metaverse is user-center by design, and human users are always the core. In this work, we introduce a multi-user VR computation offloading over wireless communication scenario. In addition, we devised a novel user-centered deep reinforcement learning approach to find a near-optimal solution. Extensive experiments demonstrate that our approach can lead to remarkable results under various requirements and constraints.
BibTeX
@inproceedings{yu2023virtual,
  author = {Wenhan Yu and Terence Jie Chua and Jun Zhao},
  title = {{Virtual Reality in Metaverse over Wireless Networks with User-centered Deep Reinforcement Learning}},
  booktitle = {ICC},
  year = {2023},
  url = {https://openreview.net/forum?id=XUiIvhWQz2},
}
Figure