Play to Earn in the Metaverse with Mobile Edge Computing over Wireless Networks: A Deep Reinforcement Learning Approach

Terence Jie Chua, Wenhan Yu, Jun ZhaoarXiv 2023 · OpenReview

Abstract

The Metaverse play-to-earn (P2E) games have been gaining popularity as they enable players to earn in-game tokens which can be translated to real-world profits. With the advancements in augmented reality (AR) technologies, users can play AR games in the Metaverse. However, these high-resolution games are compute-intensive, and in-game graphical scenes need to be offloaded from mobile devices to an edge server for computation. In this work, we consider an optimization problem where the Metaverse Service Provider (MSP)'s objective is to reduce downlink transmission latency of in-game graphics, the latency of uplink data transmission, and the worst-case (greatest) battery charge expenditure of user equipments (UEs), while maximizing the worst-case (lowest) UE resolution-influenced in-game earning potential through optimizing the downlink UE-Metaverse Base Station (UE-MBS) assignment and the uplink transmission power selection. The downlink and uplink transmissions are then executed asynchronously. We propose a multi-agent, loss-sharing (MALS) reinforcement learning model to tackle the asynchronous and asymmetric problem. We then compare the MALS model with other baseline models and show its superiority over other methods. Finally, we conduct multi-variable optimization weighting analyses and show the viability of using our proposed MALS algorithm to tackle joint optimization problems.

Introduction

The rise of the Metaverse has popularized Play-to-Earn (P2E) games, where players earn digital assets with real-world value. However, high-quality Augmented Reality (AR) experiences are compute-intensive, often exceeding the capabilities of mobile devices. To maintain smooth gameplay and high earning potential—which is directly tied to graphics resolution—devices must offload computation to Mobile Edge Computing (MEC) servers. This creates a complex resource allocation challenge involving both downlink (DL) and uplink (UL) wireless transmissions.

System Model and Problem Formulation

The system considers a multi-user environment where User Equipments (UEs) interact with multiple Metaverse Base Stations (MBSs). The optimization problem is multi-faceted and sequential:

  1. Downlink (DL): The system must decide which MBS serves which UE (discrete selection) to minimize latency and maximize graphics quality.
  2. Uplink (UL): Users must transmit their actions back to the server, requiring continuous power management to avoid draining batteries while minimizing transmission delay.
System model for asynchronous uplink and downlink optimization in a P2E Metaverse environment.

This problem is particularly challenging because it is asynchronous (DL and UL happen at different times) and asymmetric (DL involves discrete choices, while UL involves continuous variables). Standard optimization techniques struggle with these non-convex, sequential complexities. Furthermore, the MSP aims for "worst-case" optimization to ensure fairness, prioritizing the users with the lowest earning potential and highest battery drain.

Proposed Methodology: Multi-Agent Loss-Sharing (MALS)

To address these challenges, the authors propose the Multi-Agent Loss-Sharing (MALS) reinforcement learning model, built upon the Proximal Policy Optimization (PPO) algorithm.

Key Features of MALS:

  • Asymmetric Agents: The framework employs two distinct actors: a DL agent for discrete MBS selection and a UL agent for continuous power allocation.
  • Multi-Head Critic: MALS uses a centralized critic with two "heads," providing actor-centric guidance for each agent while maintaining a global view of the system state.
  • Loss-Sharing Mechanism: To ensure cooperation, the loss values from both critic heads are weighted and summed. This forces the agents to find a jointly optimal solution rather than competing for resources.
  • Reward Decomposition: The DL agent’s reward includes a weighted component of the UL reward, ensuring that initial station assignments account for subsequent uplink transmission costs and battery consumption.

Experimental Setup

The study utilizes a custom simulation environment representing a 100m x 100m area with 4 MBSs and 6 to 8 UEs. The channel model incorporates Rician fading to simulate real-world small-scale signal fluctuations. P2E in-game graphics sizes range from 800 to 1000 Mb, and a mobility model allows UEs to move up to 10 meters between iterations.

Results and Discussion

The MALS model was evaluated against two baselines: Independent Dual-Agent (IDA) and Centralized Training and Decentralized Execution (CTDE).

  • Performance and Stability: MALS consistently achieved higher cumulative rewards and faster convergence. While IDA often suffered from stability issues where one agent's needs were prioritized over the other, MALS remained stable due to its shared loss architecture.
  • Efficiency: MALS significantly reduced average downlink and uplink delays compared to random allocation and outperformed CTDE in complex scenarios (e.g., the 8-UE configuration).
  • Flexibility: Weighting analysis demonstrated that the MSP can fine-tune the system to prioritize either extreme battery conservation or maximum player profitability by adjusting utility function parameters.

Conclusion

This work introduces a robust DRL framework for optimizing the complex, asynchronous resource demands of P2E Metaverse applications. By utilizing a multi-agent loss-sharing approach, the MALS model effectively balances the trade-offs between latency, energy efficiency, and user earning potential. Future work will explore the scalability of this approach for larger deployments and more dynamic mobility scenarios.

Summary updated

Abstract

The Metaverse play-to-earn games have been gaining popularity as they enable players to earn in-game tokens which can be translated to real-world profits. With the advancements in augmented reality (AR) technologies, users can play AR games in the Metaverse. However, these high-resolution games are compute-intensive, and in-game graphical scenes need to be offloaded from mobile devices to an edge server for computation. In this work, we consider an optimization problem where the Metaverse Service Provider (MSP)'s objective is to reduce downlink transmission latency of in-game graphics, the latency of uplink data transmission, and the worst-case (greatest) battery charge expenditure of user equipments (UEs), while maximizing the worst-case (lowest) UE resolution-influenced in-game earning potential through optimizing the downlink UE-Metaverse Base Station (UE-MBS) assignment and the uplink transmission power selection. The downlink and uplink transmissions are then executed asynchronously. We propose a multi-agent, loss-sharing (MALS) reinforcement learning model to tackle the asynchronous and asymmetric problem. We then compare the MALS model with other baseline models and show its superiority over other methods. Finally, we conduct multi-variable optimization weighting analyses and show the viability of using our proposed MALS algorithm to tackle joint optimization problems.
BibTeX
@misc{chua2023play,
  author = {Terence Jie Chua and Wenhan Yu and Jun Zhao},
  title = {{Play to Earn in the Metaverse with Mobile Edge Computing over Wireless Networks: A Deep Reinforcement Learning Approach}},
  year = {2023},
  eprint = {2303.10289},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2303.10289},
}
Figure