Multiagent Deep Reinforcement Learning for Dynamic Avatar Migration in AIoT-Enabled Vehicular Metaverses With Trajectory Prediction

Junlong Chen, Jiawen Kang, Minrui Xu, Zehui Xiong, Dusit Niyato, Chuan Chen, Abbas Jamalipour, Shengli Xie

TL;DR

This paper introduces a framework for the "Vehicular Metaverse," where resource-heavy digital avatars are seamlessly migrated between roadside units (RSUs) as a car moves. By combining LSTM-based trajectory prediction with a novel Multi-Agent Deep Reinforcement Learning (MADRL) algorithm called Hybrid-MAPPO, the researchers reduced service latency by up to 30%, ensuring a smooth, immersive experience for drivers and passengers.

What Problem Does This Paper Address?

Imagine driving through a city while a high-fidelity 3D avatar acts as your AR navigator and tour guide. This "Vehicular Metaverse" (AeVeM) requires massive computational power for real-time rendering and reasoning—power that most vehicles simply don't have on board. The solution is to offload these tasks to RoadSide Units (RSUs) or cloud servers.

However, vehicles move fast. As you drive out of range of one RSU and into another, your avatar’s "brain" must migrate to the next unit to maintain low latency. This is incredibly difficult because:

  1. High Mobility: Predicting exactly where a car will be and which RSU it will talk to is a moving target.
  2. Dynamic Workloads: RSUs have limited resources and their workloads change constantly based on traffic.
  3. Heterogeneity: RSUs aren't spread out evenly; urban areas are dense, while highways are sparse.

The goal is to decide when, where, and how much of an avatar task to migrate to minimize latency and prevent "break-in-presence."

Methods

The authors propose a two-stage solution:

  1. Coverage-aware Trajectory Prediction: They use a Long Short-Term Memory (LSTM) network to predict future vehicle routes based on historical data. Crucially, this isn't just about coordinates; the model predicts the future workload of destination RSUs by estimating how many vehicles will enter their coverage area.
  2. Hybrid-MAPPO Algorithm: The migration problem is formulated as a Partially Observable Markov Decision Process (POMDP). To solve it, they extended the Multi-Agent Proximal Policy Optimization (MAPPO) algorithm to handle a hybrid action space.
  3. Discrete Actions: Choosing which RSU to migrate to (especially in dense urban areas).
  4. Continuous Actions: Deciding the portion of the avatar task to pre-migrate.

This "Hybrid-MAPPO" uses a centralized training but decentralized execution (CTDE) architecture, allowing multiple vehicles (agents) to learn collaborative migration strategies that avoid overloading specific RSUs.

Datasets

The researchers validated their model using a real-world dataset of mobility traces from 664 vehicles in Shenzhen, China. They focused on a specific urban window (latitude/longitude) to simulate realistic traffic patterns and RSU distributions. The LSTM was trained on these traces to predict future positions with high precision.

Key Insights and Ideas

  • Pre-migration is Key: Instead of waiting for a connection to drop, the system proactively moves parts of the avatar task to the next RSU. This "warm-up" significantly reduces the transition lag.
  • Workload over Distance: The "aha moment" here is that the closest RSU isn't always the best. If the nearest RSU is congested, the MADRL agents learn to offload to a slightly further RSU or even the cloud to achieve better overall latency.
  • The Hybrid Advantage: Most RL algorithms struggle with mixed action types. By splitting the policy into discrete and continuous actors that share a common value network, the agents can make more nuanced decisions (e.g., "I will move to RSU #3 AND I will send 67% of my data now").

Key Results

  • Prediction Accuracy: The LSTM model achieved a remarkably low Mean Squared Error (MSE) of $6.3 \times 10^{-5}$ for trajectory prediction.
  • Latency Reduction: The proposed framework reduced execution latency by 25% without prediction and 30% with prediction compared to baseline methods.
  • Superiority over Baselines: Hybrid-MAPPO significantly outperformed "Greedy" strategies (which only look at immediate rewards) and "Full Migration" strategies (which often overload RSUs).
  • Robustness: The system maintained its edge across varying task sizes (50MB to 250MB) and different wireless bandwidths, proving it can handle the "bursty" nature of metaverse data.

Limitations and Future Work

While the paper provides a robust technical foundation for avatar migration, a few areas remain open for exploration:

  • Security and Privacy: Migrating personal avatar data between public RSUs raises significant privacy concerns. The authors suggest that future iterations might incorporate blockchain or other privacy-preserving AI techniques.
  • Energy Efficiency: The paper focuses primarily on latency. However, in a vehicular context, the energy cost of constant data migration and high-intensity DRL computation is a critical factor that needs optimization.
  • Extreme Mobility: The current model was tested on urban traces; its performance in high-speed highway scenarios with very frequent handovers remains to be seen.

Summary updated

Abstract

Avatars, as promising digital assistants in Vehicular Metaverses, can enable drivers and passengers to immerse in 3-D virtual spaces, serving as a practical emerging example of Artificial Intelligence of Things (AIoT) in intelligent vehicular environments. The immersive experience is achieved through seamless human–avatar interaction, e.g., augmented reality navigation, which requires intensive resources that are inefficient and impractical to process on intelligent vehicles locally. Fortunately, offloading avatar tasks to roadside units (RSUs) or cloud servers for remote execution can effectively reduce resource consumption. However, the high mobility of vehicles, the dynamic workload of RSUs, and the heterogeneity of RSUs pose novel challenges to making avatar migration decisions. To address these challenges, in this article, we propose a dynamic migration framework for avatar tasks based on real-time trajectory prediction and multiagent deep reinforcement learning (MADRL). Specifically, we propose a model to predict the future trajectories of intelligent vehicles based on their historical data, indicating the future workloads of RSUs. Based on the expected workloads of RSUs, we formulate the avatar task migration problem as a long-term mixed-integer programming problem. To tackle this problem efficiently, the problem is transformed into a partially observable Markov decision process (POMDP) and solved by multiple DRL agents with hybrid continuous and discrete actions in decentralized. Numerical results demonstrate that our proposed algorithm can effectively reduce the latency of executing avatar tasks by around 25% without prediction and 30% with prediction and enhance user immersive experiences in the AIoT-enabled Vehicular Metaverse (AeVeM).
BibTeX
@article{chen2024multiagent,
  author = {Junlong Chen and Jiawen Kang and Minrui Xu and Zehui Xiong and Dusit Niyato and Chuan Chen and Abbas Jamalipour and Shengli Xie},
  title = {{Multiagent Deep Reinforcement Learning for Dynamic Avatar Migration in AIoT-Enabled Vehicular Metaverses With Trajectory Prediction}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  url = {https://openreview.net/forum?id=A6-tppzOZIv},
}

Loading related content...

Figure