User-centric Heterogeneous-action Deep Reinforcement Learning for Virtual Reality in the Metaverse over Wireless Networks

Wenhan Yu, Terence Jie Chua, Jun ZhaoIEEE Trans. Wireless Communications 2023 · DOI · OpenReview

Abstract

The Metaverse is emerging as maturing technologies are empowering the different facets. Virtual Reality (VR) technologies serve as the backbone of the virtual universe within the Metaverse to offer a highly immersive user experience. As mobility is emphasized in the Metaverse context, VR devices reduce their weights at the sacrifice of local computation abilities. In this paper, for a system consisting of a Metaverse server and multiple VR users, we consider two cases of (i) the server generating frames and transmitting them to users, and (ii) users generating frames locally and thus consuming device energy. Moreover, in our multi-user VR scenario for the Metaverse, users have different characteristics and demands for Frames Per Second (FPS). Then the channel access arrangement (including the decisions on frame generation location), and transmission powers for the downlink communications from the server to the users are jointly optimized to improve the utilities of users. This joint optimization is addressed by deep reinforcement learning (DRL) with heterogeneous actions. Our proposed user-centric DRL algorithm is called User-centric Critic with Heterogenous Actors (UCHA). Extensive experiments demonstrate that our UCHA algorithm leads to remarkable results under various requirements and constraints.

Introduction

The vision of the Metaverse relies heavily on high-fidelity Virtual Reality (VR) to provide immersive experiences. However, high-end VR rendering is computationally intensive. To maintain mobility and comfort, VR headsets are often designed to be lightweight, which limits their local processing power. This creates a critical trade-off between two operational modes: Server Generation, where the server renders frames and transmits them over wireless channels (consuming bandwidth and introducing latency), and Local Generation, where the headset renders frames itself (consuming battery and often resulting in lower resolution).

In multi-user environments, this problem is further complicated by user diversity. Different users have varying requirements for Frames Per Second (FPS), graphical resolution, and battery constraints. Managing these diverse needs requires a sophisticated resource allocation strategy that can handle both discrete decisions (channel assignment and offloading) and continuous variables (transmission power).

System Model and Problem Formulation

The system considers a Metaverse server serving multiple Virtual Users (VUs) over a Non-Orthogonal Multiple Access (NOMA) downlink. As illustrated in Figure 1, the system must decide for each frame whether it is generated at the server or locally at the VU.

System model of the multi-user VR scenario in the Metaverse, illustrating server-generated vs. local-generated frame cases and reward evaluation.
  • Case 1 (Server-generated): Frames are rendered at the server at high resolution (e.g., 1440p) and transmitted. Success depends on execution time and downlink delay staying within acceptable limits.
  • Case 2 (VU-generated): Frames are rendered locally. This consumes energy and may result in lower resolutions (e.g., 720p) depending on the device's computing power. Penalties are applied based on battery state and resolution quality.

The objective is to jointly optimize channel arrangement and power allocation to maximize a multi-faceted reward function that accounts for frame loss, resolution, and energy consumption across all users.

Methodology: The UCHA Framework

To address this complex optimization, the authors propose User-centric Critic with Heterogeneous Actors (UCHA). This Multi-Agent Deep Reinforcement Learning (MADRL) architecture is built on two primary innovations:

1. Heterogeneous Actors

Instead of a single massive action space, UCHA employs two asymmetric actors to handle different types of control variables. Actor 1 optimizes discrete actions, specifically channel access and the decision to offload or compute locally. Actor 2 optimizes continuous actions, namely the downlink power allocation. This separation allows for more efficient training and better handling of the hybrid action space.

2. User-centric Critic

Traditional DRL uses a single global critic to evaluate the total reward. UCHA adopts a multi-head critic inspired by Hybrid Reward Architecture (HRA). As shown in Figure 2, the critic evaluates the state-value for each VU individually. This "user-centric" view prevents the "lazy agent" problem, where the system might sacrifice one user's experience to maximize the global average. By tracking individual rewards, the agent ensures fairness and better satisfies specific user constraints.

The UCHA framework architecture, featuring heterogeneous actors for discrete and continuous actions and a user-centric critic for individual reward evaluation.

Experimental Results

The UCHA algorithm was evaluated in a simulated 30x30m indoor environment with 5 to 8 VR users. Key findings include:

  • Convergence and Efficiency: In a 6-user scenario, UCHA reached peak performance approximately five times faster than standard baselines like HAPPO. This is attributed to the reward decomposition and the use of an early termination penalty for failing FPS requirements, which provides denser feedback during training.
  • Reliability and Fairness: UCHA was the only tested algorithm capable of consistently meeting FPS targets for all users in high-density (8-user) scenarios. It significantly improved the "Worst VU" metric, ensuring that even the most disadvantaged user (e.g., one with low battery or poor channel conditions) maintained a tolerable experience.
  • Intelligent Resource Management: The model successfully learned to offload users with high FPS demands to the server while requiring users with sufficient battery and local compute power to render locally, effectively balancing the load on the wireless spectrum.

Conclusion

UCHA provides a robust framework for managing the heterogeneous demands of VR in the Metaverse. By decomposing rewards on a per-user basis and utilizing heterogeneous actors, the system achieves superior performance and fairness compared to traditional DRL approaches. Future work will explore the impact of high-speed user mobility and the integration of dynamic compression ratios to further optimize bandwidth usage.

Summary updated

Abstract

The Metaverse is emerging as maturing technologies are empowering the different facets. Virtual Reality (VR) technologies serve as the backbone of the virtual universe within the Metaverse to offer a highly immersive user experience. As mobility is emphasized in the Metaverse context, VR devices reduce their weights at the sacrifice of local computation abilities. In this paper, for a system consisting of a Metaverse server and multiple VR users, we consider two cases of (i) the server generating frames and transmitting them to users, and (ii) users generating frames locally and thus consuming device energy. Moreover, in our multi-user VR scenario for the Metaverse, users have different characteristics and demands for Frames Per Second (FPS). Then the channel access arrangement (including the decisions on frame generation location), and transmission powers for the downlink communications from the server to the users are jointly optimized to improve the utilities of users. This joint optimization is addressed by deep reinforcement learning (DRL) with heterogeneous actions. Our proposed user-centric DRL algorithm is called User-centric Critic with Heterogenous Actors (UCHA). Extensive experiments demonstrate that our UCHA algorithm leads to remarkable results under various requirements and constraints.
BibTeX
@article{yu2023user,
  author = {Wenhan Yu and Terence Jie Chua and Jun Zhao},
  title = {{User-centric Heterogeneous-action Deep Reinforcement Learning for Virtual Reality in the Metaverse over Wireless Networks}},
  journal = {IEEE Transactions on Wireless Communications},
  year = {2023},
  doi = {10.1109/twc.2023.3277226},
}
Figure