Abstract
Precise pose estimation is fundamental to the realism of Mobile Augmented Reality (MAR), requiring accurate 3D maps of environmental feature points. However, the computational demands of map management and the variability of wireless uplink bandwidth present significant bottlenecks. This paper proposes a Digital Twin (DT)-based 3D map management scheme that utilizes Model-Based Reinforcement Learning (MBRL) to optimize frame uploading. By predicting future camera frames and simulating artificial experiences, the system minimizes pose uncertainty under limited bandwidth. Our results show that the proposed Adaptive Map Management (AMM) algorithm achieves faster convergence and higher pose accuracy than conventional methods.
Introduction
For Augmented Reality (AR) to be effective, virtual content must be seamlessly registered with the physical environment. This relies on precise pose estimation—the continuous determination of the user's position and orientation. While edge-assisted MAR offloads the heavy lifting of 3D mapping to remote servers, the limited and dynamic nature of wireless uplink data rates (e.g., switching between mmWave and sub-6 GHz) often prevents the transmission of every camera frame. Existing "myopic" strategies, such as uploading only the latest frame or using fixed intervals, fail to account for user mobility and network fluctuations, leading to increased pose uncertainty.
System Architecture and Methodology
The proposed system, Adaptive Map Management (AMM), introduces a Digital Twin (DT) on the edge server to mirror the MAR device's state.
Digital Twin and Predictive Modeling
The DT employs a Recurrent Neural Network (RNN) with Long Short-Term Memory (LSTM) layers to predict the 3D map points of future camera frames based on historical movement data. This predictive capability allows the system to simulate potential future states, effectively "hallucinating" the geometric impact of future frame selections.
Model-Based Reinforcement Learning (MBRL)
The core of the AMM algorithm is an MBRL agent that learns an optimal policy for frame uploading ($U_t$) and map pruning ($C_t$). The 3D map is represented as an undirected graph using a Graph Convolutional Network (GCN), where nodes represent frames and edges denote co-visibility of landmarks. The Laplacian matrix of this graph is used to quantify pose estimation uncertainty.
Unlike model-free RL, which relies solely on real-world interactions, the MBRL approach uses the DT to generate "artificial experiences." This allows the agent to train on both real and emulated data, significantly improving data efficiency and adaptation speed.
Experimental Evaluation
The scheme was evaluated using the SUN3D dataset ("westgate-playroom") and the ORB-SLAM framework for ground-truth extraction. Network dynamics were modeled using a two-state Markov chain to simulate transitions between high-rate and low-rate states.
Key Results
The AMM scheme consistently outperformed "Latest Frame First" (LFF) and "Periodical Uploading" (PU) benchmarks, particularly in scenarios where high-speed network availability was limited (40-60%). By prioritizing geometrically diverse frames, the system maintained lower pose uncertainty.
As shown in the convergence analysis, the MBRL approach reaches a stable, low-uncertainty policy much faster than standard model-free RL. The use of artificial experiences (represented by $N$ iterations in the AMM algorithm) effectively guides the learning process, reducing the number of real-world episodes required for optimization.
Conclusion
This research demonstrates that integrating a Digital Twin with Model-Based Reinforcement Learning can significantly enhance 3D map management for edge-assisted MAR. By predicting future states and optimizing frame selection, the AMM algorithm ensures robust pose estimation even under constrained network conditions. Future work will explore the computational trade-offs of server-side GCN/LSTM processing and extend the framework to multi-user collaborative mapping environments.
Summary updated