Multi-Agent Deep Reinforcement Learning for Dynamic Avatar Migration in AIoT-enabled Vehicular Metaverses with Trajectory Prediction

Junlong Chen, Jiawen Kang, Minrui Xu, Zehui Xiong, Dusit Niyato, Chuan Chen, Abbas Jamalipour, Shengli Xie

Abstract

Avatars, as promising digital assistants in Vehicular Metaverses, can enable drivers and passengers to immerse in 3D virtual spaces, serving as a practical emerging example of Artificial Intelligence of Things (AIoT) in intelligent vehicular environments. The immersive experience is achieved through seamless human-avatar interaction, e.g., augmented reality navigation, which requires intensive resources that are inefficient and impractical to process on intelligent vehicles locally. Fortunately, offloading avatar tasks to RoadSide Units (RSUs) or cloud servers for remote execution can effectively reduce resource consumption. However, the high mobility of vehicles, the dynamic workload of RSUs, and the heterogeneity of RSUs pose novel challenges to making avatar migration decisions. To address these challenges, in this paper, we propose a dynamic migration framework for avatar tasks based on real-time trajectory prediction and Multi-Agent Deep Reinforcement Learning (MADRL). Specifically, we propose a model to predict the future trajectories of intelligent vehicles based on their historical data, indicating the future workloads of RSUs. Based on the expected workloads of RSUs, we formulate the avatar task migration problem as a long-term mixed integer programming problem. To tackle this problem efficiently, the problem is transformed into a Partially Observable Markov Decision Process (POMDP) and solved by multiple DRL agents with hybrid continuous and discrete actions in decentralized. Numerical results demonstrate that our proposed algorithm can effectively reduce the latency of executing avatar tasks by around 25% without prediction and 30% with prediction and enhance user immersive experiences in the AIoT-enabled Vehicular Metaverse (AeVeM).

Introduction

The "Vehicular Metaverse" (AeVeM) represents a paradigm shift in intelligent transportation, where high-fidelity 3D avatars serve as interactive digital assistants for AR navigation and entertainment. However, the computational demands for real-time rendering and reasoning exceed the capabilities of most on-board vehicle units. Offloading these tasks to RoadSide Units (RSUs) or cloud servers is necessary, but the high mobility of vehicles necessitates frequent migration of avatar "brains" to maintain low latency and prevent "break-in-presence."

System model for dynamic avatar migration in the Vehicular Metaverse, illustrating trajectory prediction and pre-migration between RSUs.

Proposed Framework

The proposed solution addresses the migration challenge through a two-stage approach:

1. Coverage-aware Trajectory Prediction

Using a Long Short-Term Memory (LSTM) network, the system predicts future vehicle routes based on historical mobility traces. This prediction is not merely spatial; it allows the system to estimate the future workload of destination RSUs by calculating the expected number of vehicles entering their coverage areas.

2. Hybrid-MAPPO Algorithm

The migration decision-making is modeled as a Partially Observable Markov Decision Process (POMDP). To handle the complexities of choosing a destination RSU (discrete action) and determining the portion of the task to pre-migrate (continuous action), the researchers developed Hybrid-MAPPO. This algorithm extends Multi-Agent Proximal Policy Optimization to a hybrid action space.

The Hybrid-MAPPO architecture, featuring discrete and continuous actors to handle complex migration decisions.

The architecture utilizes Centralized Training but Decentralized Execution (CTDE), enabling multiple vehicle agents to learn collaborative strategies that prevent RSU congestion while minimizing individual latency.

Experimental Evaluation

The framework was validated using real-world mobility traces from 664 vehicles in Shenzhen, China. Key findings include:

  • Prediction Precision: The LSTM model achieved a remarkably low Mean Squared Error (MSE) of $6.3 \times 10^{-5}$ for trajectory prediction.
  • Latency Performance: The framework reduced service latency by 25% without prediction and 30% when trajectory prediction was integrated.
  • Robustness: Hybrid-MAPPO outperformed greedy and full-migration baselines across various task sizes (50MB to 250MB) and bandwidth conditions.

Conclusion

By combining proactive trajectory prediction with a hybrid MADRL approach, this framework ensures seamless avatar migration in dynamic vehicular environments. Future work will focus on enhancing data privacy through techniques like blockchain and optimizing the energy efficiency of the migration process.

Summary updated

Abstract

Avatars, as promising digital assistants in Vehicular Metaverses, can enable drivers and passengers to immerse in 3D virtual spaces, serving as a practical emerging example of Artificial Intelligence of Things (AIoT) in intelligent vehicular environments. The immersive experience is achieved through seamless human-avatar interaction, e.g., augmented reality navigation, which requires intensive resources that are inefficient and impractical to process on intelligent vehicles locally. Fortunately, offloading avatar tasks to RoadSide Units (RSUs) or cloud servers for remote execution can effectively reduce resource consumption. However, the high mobility of vehicles, the dynamic workload of RSUs, and the heterogeneity of RSUs pose novel challenges to making avatar migration decisions. To address these challenges, in this paper, we propose a dynamic migration framework for avatar tasks based on real-time trajectory prediction and Multi-Agent Deep Reinforcement Learning (MADRL). Specifically, we propose a model to predict the future trajectories of intelligent vehicles based on their historical data, indicating the future workloads of RSUs.Based on the expected workloads of RSUs, we formulate the avatar task migration problem as a long-term mixed integer programming problem. To tackle this problem efficiently, the problem is transformed into a Partially Observable Markov Decision Process (POMDP) and solved by multiple DRL agents with hybrid continuous and discrete actions in decentralized. Numerical results demonstrate that our proposed algorithm can effectively reduce the latency of executing avatar tasks by around 25% without prediction and 30% with prediction and enhance user immersive experiences in the AIoT-enabled Vehicular Metaverse (AeVeM).
BibTeX
@article{chen2024multi,
  author = {Junlong Chen and Jiawen Kang and Minrui Xu and Zehui Xiong and Dusit Niyato and Chuan Chen and Abbas Jamalipour and Shengli Xie},
  title = {{Multi-Agent Deep Reinforcement Learning for Dynamic Avatar Migration in AIoT-enabled Vehicular Metaverses with Trajectory Prediction}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  url = {https://openreview.net/forum?id=A6-tppzOZIv},
}
Figure