Digital Twin-Based 3D Map Management for Edge-Assisted Mobile Augmented Reality

Conghao Zhou, Jie Gao, Mushu Li, Nan Cheng, Xuemin Shen, Weihua ZhuangarXiv 2023 · DOI · arXiv

Abstract

Precise pose estimation is fundamental to the realism of Mobile Augmented Reality (MAR), requiring accurate 3D maps of environmental feature points. However, the computational demands of map management and the variability of wireless uplink bandwidth present significant bottlenecks. This paper proposes a Digital Twin (DT)-based 3D map management scheme that utilizes Model-Based Reinforcement Learning (MBRL) to optimize frame uploading. By predicting future camera frames and simulating artificial experiences, the system minimizes pose uncertainty under limited bandwidth. Our results show that the proposed Adaptive Map Management (AMM) algorithm achieves faster convergence and higher pose accuracy than conventional methods.

Introduction

For Augmented Reality (AR) to be effective, virtual content must be seamlessly registered with the physical environment. This relies on precise pose estimation—the continuous determination of the user's position and orientation. While edge-assisted MAR offloads the heavy lifting of 3D mapping to remote servers, the limited and dynamic nature of wireless uplink data rates (e.g., switching between mmWave and sub-6 GHz) often prevents the transmission of every camera frame. Existing "myopic" strategies, such as uploading only the latest frame or using fixed intervals, fail to account for user mobility and network fluctuations, leading to increased pose uncertainty.

System Architecture and Methodology

The proposed system, Adaptive Map Management (AMM), introduces a Digital Twin (DT) on the edge server to mirror the MAR device's state.

System architecture of the edge-assisted MAR with Digital Twin-based map management.

Digital Twin and Predictive Modeling

The DT employs a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) layers to predict the 3D map points of future camera frames based on historical movement data. This predictive capability allows the system to simulate potential future states, effectively "hallucinating" the geometric impact of future frame selections.

Model-Based Reinforcement Learning (MBRL)

The core of the AMM algorithm is an MBRL agent that learns an optimal policy for frame uploading ($U_t$) and map pruning ($C_t$). The 3D map is represented as an undirected graph using a Graph Convolutional Network (GCN), where nodes represent frames and edges denote co-visibility of landmarks. The Laplacian matrix of this graph is used to quantify pose estimation uncertainty.

Unlike model-free RL, which relies solely on real-world interactions, the MBRL approach uses the DT to generate "artificial experiences." This allows the agent to train on both real and emulated data, significantly improving data efficiency and adaptation speed.

Experimental Evaluation

The scheme was evaluated using the SUN3D dataset ("westgate-playroom") and the ORB-SLAM framework for ground-truth extraction. Network dynamics were modeled using a two-state Markov chain to simulate transitions between high-rate and low-rate states.

Key Results

The AMM scheme consistently outperformed "Latest Frame First" (LFF) and "Periodical Uploading" (PU) benchmarks, particularly in scenarios where high-speed network availability was limited (40-60%). By prioritizing geometrically diverse frames, the system maintained lower pose uncertainty.

Convergence performance of the proposed MBRL scheme compared to model-free RL.

As shown in the convergence analysis, the MBRL approach reaches a stable, low-uncertainty policy much faster than standard model-free RL. The use of artificial experiences (represented by $N$ iterations in the AMM algorithm) effectively guides the learning process, reducing the number of real-world episodes required for optimization.

Conclusion

This research demonstrates that integrating a Digital Twin with Model-Based Reinforcement Learning can significantly enhance 3D map management for edge-assisted MAR. By predicting future states and optimizing frame selection, the AMM algorithm ensures robust pose estimation even under constrained network conditions. Future work will explore the computational trade-offs of server-side GCN/LSTM processing and extend the framework to multi-user collaborative mapping environments.

Summary updated

Abstract

In this paper, we design a 3D map management scheme for edge-assisted mobile augmented reality (MAR) to support the pose estimation of individual MAR device, which uploads camera frames to an edge server. Our objective is to minimize the pose estimation uncertainty of the MAR device by periodically selecting a proper set of camera frames for uploading to update the 3D map. To address the challenges of the dynamic uplink data rate and the time-varying pose of the MAR device, we propose a digital twin (DT)-based approach to 3D map management. First, a DT is created for the MAR device, which emulates 3D map management based on predicting subsequent camera frames. Second, a model-based reinforcement learning (MBRL) algorithm is developed, utilizing the data collected from both the actual and the emulated data to manage the 3D map. With extensive emulated data provided by the DT, the MBRL algorithm can quickly provide an adaptive map management policy in a highly dynamic environment. Simulation results demonstrate that the proposed DT-based 3D map management outperforms benchmark schemes by achieving lower pose estimation uncertainty and higher data efficiency in dynamic environments.
BibTeX
@misc{zhou2023digital,
  author = {Conghao Zhou and Jie Gao and Mushu Li and Nan Cheng and Xuemin Shen and Weihua Zhuang},
  title = {{Digital Twin-Based 3D Map Management for Edge-Assisted Mobile Augmented Reality}},
  year = {2023},
  eprint = {2305.16571},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2305.16571},
}
Figure