Verifiable Access Control for Augmented Reality Localization and Mapping

Shaowei Zhu, Hyo Jin Kim, Maurizio Monge, G. Edward Suh, Armin Alaghi, Brandon Reagen, Vincent T. LeearXiv 2022 · DOI · arXiv

Introduction

Augmented Reality (AR) relies on the continuous creation and querying of high-fidelity 3D maps. These maps, typically represented as sparse point clouds, allow devices to determine their precise location (localization) and understand the geometry of their surroundings (mapping). However, these digital representations are "privacy goldmines." A stolen map can be used to reconstruct the detailed layout of a private residence or expose sensitive personal belongings. Furthermore, the lack of robust access control allows malicious actors to "vandalize" virtual spaces by injecting fraudulent data or deleting existing map points, compromising the integrity of the AR experience.

VACMaps introduces a first-of-its-kind access control system designed to protect both the privacy and integrity of these 3D maps. By leveraging a domain-specific language and formal verification via Satisfiability Modulo Theories (SMT) solvers, VACMaps enables users to define and mathematically prove permissions for mapping, modifying, or localizing within specific physical boundaries.

System Architecture and the AR Pipeline

To understand the necessity of VACMaps, one must consider the standard AR workflow. As illustrated in Figure 1, the process is split between device-side processing and server-side management.

Overview of the standard AR localization and mapping pipeline between device and server.

In this pipeline, a user's device extracts visual descriptors from the environment to generate submaps or issue localization requests. These are sent to a server which maintains a global Map Database. Without a dedicated security layer, any user with network access could potentially read from or write to this database for any location. VACMaps inserts itself into this flow, acting as a gatekeeper that validates every request against a set of spatial policies.

VACMaps: Verifiable Access Control

The core of VACMaps is its ability to provide fine-grained, verifiable permissions. The system architecture, detailed in Figure 2, consists of three primary components: the VMAC Language, a spatial hierarchy, and a formal reasoning engine.

The VACMaps access control workflow, utilizing formal reasoning to validate requests against spatial policies.

VMAC Lang and Spatial Hierarchy

The researchers developed VMAC Lang, a Domain-Specific Language (DSL) that allows owners to write human-readable policies. A policy specifies a Principal (the actor), an Action (read, write, or localize), a Space (the 3D volume), and optional Conditions (such as time-of-day or physical proximity).

To manage these spaces efficiently, VACMaps organizes the physical world into a hierarchy (e.g., Building → Floor → Room). Policies are inherited down this tree, and the system employs a "deny overrides allow" logic to resolve any logical conflicts. This hierarchical approach is paired with 3D segment trees for spatial indexing, allowing the server to instantly identify which policies apply to a specific coordinate, even in dense environments with thousands of overlapping zones.

Formal Reasoning with SMT

Unlike traditional permission systems that rely on simple "if-else" logic, VACMaps translates policies into SMT formulas. When an access request is received, the system uses the Z3 SMT solver to determine if the request is logically permitted. This formal approach allows for automated auditing; users can query the system to ensure that a new policy doesn't accidentally grant unauthorized access or that a specific "deny" rule remains effective against broader "allow" rules.

Evaluation and Results

The system was evaluated using a synthetically generated dataset of a single-family home, including 3D segmentations and simulated "video tours" to mimic realistic user behavior.

Performance and Scalability

The evaluation demonstrated that VACMaps can process access requests in near real-time. While processing time scales with the number of feature points in a frame, the implementation of a Least Recently Used (LRU) cache—storing recent permission results—halved the average processing time.

Furthermore, the system exhibited massive scalability. In tests involving up to 100,000 virtual spaces, the spatial hierarchy ensured that lookup times remained stable, following logarithmic complexity. This suggests that VACMaps is capable of handling large-scale deployments, such as smart cities or expansive corporate campuses.

Discussion: The Bystander Problem and Future Work

A unique challenge addressed by VACMaps is the "bystander space" scenario. This occurs when a user in a public area (like a hallway) can visually "see" into a private area (like an office). VACMaps handles this by denying access to map points located within the private volume, regardless of the user's physical standing point.

Despite its strengths, the system faces limitations regarding estimation errors. Current SLAM and GPS technologies have inherent noise; a slight miscalculation in a user's position could lead to an incorrect denial of access at a boundary. Future work aims to incorporate "robust" policies that account for these spatial uncertainties. Additionally, while the current focus is on 3D point clouds, the framework will eventually need to expand to cover semantic meshes and persistent virtual objects.

Conclusion

VACMaps represents a significant step forward in securing the spatial web. By moving beyond physical locks and simple digital permissions to a verifiable, logic-based framework, it provides the necessary tools to protect user privacy and data integrity in the burgeoning era of Augmented Reality.

Summary updated

Abstract

Localization and mapping is a key technology for bridging the virtual and physical worlds in augmented reality (AR). Localization and mapping works by creating and querying maps made of anchor points that enable the overlay of these two worlds. As a result, information about the physical world is captured in the map and naturally gives rise to concerns around who can map physical spaces as well as who can access or modify the virtual ones. This paper discusses how we can provide access controls over virtual maps as a basic building block to enhance security and privacy of AR systems. In particular, we propose VACMaps: an access control system for localization and mapping using formal methods. VACMaps defines a domain-specific language that enables users to specify access control policies for virtual spaces. Access requests to virtual spaces are then evaluated against relevant policies in a way that preserves confidentiality and integrity of virtual spaces owned by the users. The precise semantics of the policies are defined by SMT formulas, which allow VACMaps to reason about properties of access policies automatically. An evaluation of VACMaps is provided using an AR testbed of a single-family home. We show that VACMaps is scalable in that it can run at practical speeds and that it can also reason about access control policies automatically to detect potential policy misconfigurations.
BibTeX
@misc{zhu2022verifiable,
  author = {Shaowei Zhu and Hyo Jin Kim and Maurizio Monge and G. Edward Suh and Armin Alaghi and Brandon Reagen and Vincent T. Lee},
  title = {{Verifiable Access Control for Augmented Reality Localization and Mapping}},
  year = {2022},
  eprint = {2203.13308},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2203.13308},
}
Figure