MobileARLoc: On-device Robust Absolute Localisation for Pervasive Markerless Mobile AR

Changkun Liu, Yukun Zhao, Tristan Braud

Introduction

Absolute localisation—the ability to determine a device's precise 6-degree-of-freedom (6DOF) pose within a global coordinate system—is fundamental for seamless Augmented Reality (AR). Without it, virtual objects cannot be anchored reliably to the physical world. Current solutions often force a compromise between the high accuracy of server-side structure-based methods and the low latency but high noise of on-device deep learning models. MobileARLoc bridges this gap by introducing a framework that achieves server-grade localisation robustness directly on mobile hardware.

The Challenge of Absolute Localisation

Mobile AR developers typically rely on three distinct approaches, each with significant drawbacks:

  1. Structure-based methods: These provide high accuracy but are computationally intensive, often requiring image offloading to a server. This introduces latency, jitter, and privacy risks.
  2. Absolute Pose Regressors (APRs): These lightweight deep learning models estimate pose from single images. While fast and on-device, they are notoriously noisy and prone to significant outliers.
  3. Visual-Inertial Odometry (VIO): Standard in tools like ARKit and ARCore, VIO tracks relative movement accurately in the short term but suffers from cumulative "drift," losing global alignment over time.

The MobileARLoc Framework

MobileARLoc leverages the complementary strengths of APRs and VIO. While APRs are noisy, they do not drift; conversely, VIO is smooth but drifts. The framework integrates these using a feedback loop consisting of two primary stages.

The MobileARLoc framework architecture, illustrating the feedback loop between VIO, APR, and the alignment/optimization stages.

Alignment Stage

The system identifies a sequence of $N+1$ consecutive frames where the relative movement predicted by the APR matches the movement reported by the VIO. When consistency is found, the APR predictions are considered "reliable." These poses are geometrically averaged to establish a "reference pose," effectively aligning the VIO’s local coordinate system with the global world coordinate system.

Pose Optimization Stage

Once aligned, the system monitors incoming APR predictions. If a prediction remains consistent with the VIO's trajectory, it is output directly. If the APR produces a noisy outlier, the system discards it and instead computes the pose by applying the VIO’s relative displacement to the last verified global pose. A similarity metric continuously monitors for VIO drift, triggering a return to the Alignment stage if accuracy degrades.

Evaluation and Key Results

The framework was evaluated using a custom mobile dataset collected via an iPhone 14 Pro Max across diverse indoor and outdoor environments. Ground truth was established using a COLMAP Structure-from-Motion pipeline. Two popular APR models, PoseNet (PN) and MS-Transformer (MS-T), served as the base estimators.

Translation error histograms comparing standalone APR models (PN, MS-T) against their MobileARLoc-enhanced versions (PN^vio, MS-T^vio).

The results demonstrate a significant performance leap:

  • Accuracy Improvement: For the MS-Transformer model in outdoor settings, MobileARLoc improved translation accuracy by up to 47% and rotation accuracy by 66%.
  • Outlier Suppression: The system effectively eliminated extreme pose errors (e.g., >5m or >10°), which are typically catastrophic for AR immersion.
  • Efficiency: The entire pipeline runs on-device with an inference speed of approximately 80ms (12.5 FPS), making it suitable for real-time applications without the need for server offloading.

Discussion and Future Work

MobileARLoc demonstrates that VIO can serve as a powerful real-time filter for deep-learning-based pose estimators. By focusing on inference-time refinement rather than just training-time accuracy, the system achieves a balance of speed, privacy, and precision.

However, challenges remain. Indoor environments with repetitive textures can hinder the Alignment stage, and the system requires an initial set of "good" frames to calibrate. Future work may involve integrating uncertainty estimation directly into the APR training to further improve the selection of reliable reference poses.

Summary updated

Abstract

Recent years have seen significant improvement in absolute camera pose estimation, paving the way for pervasive markerless Augmented Reality (AR). However, accurate absolute pose estimation techniques are computation- and storage-heavy, requiring computation offloading. As such, AR systems rely on visual-inertial odometry (VIO) to track the device’s relative pose between requests to the server. However, VIO suffers from drift, requiring frequent absolute repositioning. This paper introduces MobileARLoc, a new framework for on-device large-scale markerless mobile AR that combines an absolute pose regressor (APR) with a local VIO tracking system. Absolute pose regressors (APRs) provide fast on-device pose estimation at the cost of reduced accuracy. To address APR accuracy and reduce VIO drift, MobileARLoc creates a feedback loop where VIO pose estimations refine the APR predictions. The VIO system identifies reliable predictions of APR, which are then used to compensate for the VIO drift. We comprehensively evaluate MobileARLoc through dataset simulations. MobileARLoc halves the error compared to the underlying APR and achieves fast (80ms) on-device inference speed.
BibTeX
@inproceedings{liu2024mobilearloc,
  author = {Changkun Liu and Yukun Zhao and Tristan Braud},
  title = {{MobileARLoc: On-device Robust Absolute Localisation for Pervasive Markerless Mobile AR}},
  booktitle = {PerCom Workshops},
  year = {2024},
  url = {https://openreview.net/forum?id=JCzTeHLnXW},
}
Figure