Abstract
The Metaverse is emerging as maturing technologies are empowering the different facets. Virtual Reality (VR) technologies serve as the backbone of the virtual universe within the Metaverse to offer a highly immersive user experience. As mobility is emphasized in the Metaverse context, VR devices reduce their weights at the sacrifice of local computation abilities. In this paper, for a system consisting of a Metaverse server and multiple VR users, we consider two cases of (i) the server generating frames and transmitting them to users, and (ii) users generating frames locally and thus consuming device energy. Moreover, in our multi-user VR scenario for the Metaverse, users have different characteristics and demands for Frames Per Second (FPS). Then the channel access arrangement (including the decisions on frame generation location), and transmission powers for the downlink communications from the server to the users are jointly optimized to improve the utilities of users. This joint optimization is addressed by deep reinforcement learning (DRL) with heterogeneous actions. Our proposed user-centric DRL algorithm is called User-centric Critic with Heterogenous Actors (UCHA). Extensive experiments demonstrate that our UCHA algorithm leads to remarkable results under various requirements and constraints.
Introduction
The vision of the Metaverse relies heavily on high-fidelity Virtual Reality (VR) to provide immersive experiences. However, high-end VR rendering is computationally intensive. To maintain mobility and comfort, VR headsets are often designed to be lightweight, which limits their local processing power. This creates a critical trade-off between two operational modes: Server Generation, where the server renders frames and transmits them over wireless channels (consuming bandwidth and introducing latency), and Local Generation, where the headset renders frames itself (consuming battery and often resulting in lower resolution).
In multi-user environments, this problem is further complicated by user diversity. Different users have varying requirements for Frames Per Second (FPS), graphical resolution, and battery constraints. Managing these diverse needs requires a sophisticated resource allocation strategy that can handle both discrete decisions (channel assignment and offloading) and continuous variables (transmission power).
System Model and Problem Formulation
The system considers a Metaverse server serving multiple Virtual Users (VUs) over a Non-Orthogonal Multiple Access (NOMA) downlink. As illustrated in Figure 1, the system must decide for each frame whether it is generated at the server or locally at the VU.
- Case 1 (Server-generated): Frames are rendered at the server at high resolution (e.g., 1440p) and transmitted. Success depends on execution time and downlink delay staying within acceptable limits.
- Case 2 (VU-generated): Frames are rendered locally. This consumes energy and may result in lower resolutions (e.g., 720p) depending on the device's computing power. Penalties are applied based on battery state and resolution quality.
The objective is to jointly optimize channel arrangement and power allocation to maximize a multi-faceted reward function that accounts for frame loss, resolution, and energy consumption across all users.
Methodology: The UCHA Framework
To address this complex optimization, the authors propose User-centric Critic with Heterogeneous Actors (UCHA). This Multi-Agent Deep Reinforcement Learning (MADRL) architecture is built on two primary innovations:
1. Heterogeneous Actors
Instead of a single massive action space, UCHA employs two asymmetric actors to handle different types of control variables. Actor 1 optimizes discrete actions, specifically channel access and the decision to offload or compute locally. Actor 2 optimizes continuous actions, namely the downlink power allocation. This separation allows for more efficient training and better handling of the hybrid action space.
2. User-centric Critic
Traditional DRL uses a single global critic to evaluate the total reward. UCHA adopts a multi-head critic inspired by Hybrid Reward Architecture (HRA). As shown in Figure 2, the critic evaluates the state-value for each VU individually. This "user-centric" view prevents the "lazy agent" problem, where the system might sacrifice one user's experience to maximize the global average. By tracking individual rewards, the agent ensures fairness and better satisfies specific user constraints.
Experimental Results
The UCHA algorithm was evaluated in a simulated 30x30m indoor environment with 5 to 8 VR users. Key findings include:
- Convergence and Efficiency: In a 6-user scenario, UCHA reached peak performance approximately five times faster than standard baselines like HAPPO. This is attributed to the reward decomposition and the use of an early termination penalty for failing FPS requirements, which provides denser feedback during training.
- Reliability and Fairness: UCHA was the only tested algorithm capable of consistently meeting FPS targets for all users in high-density (8-user) scenarios. It significantly improved the "Worst VU" metric, ensuring that even the most disadvantaged user (e.g., one with low battery or poor channel conditions) maintained a tolerable experience.
- Intelligent Resource Management: The model successfully learned to offload users with high FPS demands to the server while requiring users with sufficient battery and local compute power to render locally, effectively balancing the load on the wireless spectrum.
Conclusion
UCHA provides a robust framework for managing the heterogeneous demands of VR in the Metaverse. By decomposing rewards on a per-user basis and utilizing heterogeneous actors, the system achieves superior performance and fairness compared to traditional DRL approaches. Future work will explore the impact of high-speed user mobility and the integration of dynamic compression ratios to further optimize bandwidth usage.
Summary updated