Digital Twin-based 3D Map Management for Edge-assisted Device Pose Tracking in Mobile AR

Conghao Zhou, Jie Gao, Mushu Li, Nan Cheng, Xuemin Shen, Weihua Zhuang

Abstract

Mobile Augmented Reality (AR) requires precise 3D pose tracking, typically implemented via Simultaneous Localization and Mapping (SLAM). However, the high computational demands of SLAM necessitate offloading map management to edge servers, which introduces challenges related to fluctuating network conditions. This paper proposes a Digital Twin (DT)-based management scheme that utilizes deep variational inference to model network dynamics and model-based reinforcement learning (MBRL) to optimize frame uploading and map pruning. By minimizing pose estimation uncertainty, the system ensures robust tracking in resource-constrained, edge-assisted environments.

Introduction

For Mobile AR (MAR) to achieve seamless immersion, virtual objects must be perfectly aligned with the physical world. This requires continuous tracking of the device's 3D pose (position and orientation). While edge-assisted SLAM mitigates local resource constraints by offloading map management, it is highly sensitive to uplink data rate fluctuations and unpredictable user movement. Traditional heuristics often fail to account for these non-stationary network dynamics, leading to tracking failures when connections become jittery.

System Architecture

The proposed system splits the SLAM process between the MAR device and an edge server. The device performs local pose calculation using a reference map, while the edge server maintains a high-fidelity 3D map and handles complex mapping updates. The system must decide in real-time which camera frames to upload and which to prune from the edge map to stay within storage limits.

Edge-assisted 3D map management architecture for mobile AR.

Methodology

The core of the solution lies in the User Digital Twin (UDT) and a model-based reinforcement learning controller designed to bridge the gap between networking and SLAM performance.

User Digital Twin (UDT)

The UDT acts as a data model of the MAR device, employing Deep Variational Inference to extract latent features of the uplink data rate. Unlike standard Markov models that assume static network behavior, the UDT can infer and emulate non-stationary network conditions where the underlying rules of the network change over time.

Model-Based Reinforcement Learning (MBRL)

The map management problem is formulated as a Bayes-adaptive Markov Decision Process (BAMDP). An RL agent, using an Actor-Critic framework, makes optimal decisions about frame uploading and map pruning. To accelerate learning, the agent utilizes Blended Experience Replay, training on a mix of real-world data and artificial experiences generated by the UDT.

The User Digital Twin (UDT) and MBRL controller framework featuring blended experience replay.

Uncertainty as a Metric

Since ground truth for a device's pose is rarely available in real-time, the system optimizes for pose estimation uncertainty. By representing the 3D map as a weighted undirected graph, the edge server calculates a unitless uncertainty score based on the reduced Laplacian matrix. This allows the server to prioritize frames that offer the best geometric coverage.

Performance Evaluation

The system was evaluated using the SUN3D dataset and the ORB-SLAM framework. Key results include:

  • Network Modeling: The UDT outperformed standard Markov and LSTM-based predictors in capturing time-varying uplink rates.
  • Tracking Performance: The DT-based scheme consistently achieved lower pose estimation uncertainty compared to benchmarks such as Latest Frame First (LFF) and Periodical Uploading (PU).
  • Robustness: The MBRL approach showed a significant advantage during high-variance network conditions, proving its suitability for unstable 5G/6G connections.

Conclusion

The integration of Digital Twins with MBRL provides a robust framework for managing edge-assisted AR services. While this work focuses on single-user scenarios, future research will address multi-user resource competition and the power consumption costs of continuous uplink transmissions.

Summary updated

BibTeX
@article{zhou2024digital,
  author = {Conghao Zhou and Jie Gao and Mushu Li and Nan Cheng and Xuemin Shen and Weihua Zhuang},
  title = {{Digital Twin-based 3D Map Management for Edge-assisted Device Pose Tracking in Mobile AR}},
  journal = {IEEE Internet of Things Journal},
  year = {2024},
  doi = {10.1109/jiot.2024.3360414},
}
Figure