Enabling Fine-Grained Permissions for Augmented Reality Applications with Recognizers

Suman Jana, David Molnar, Alexander Moshchuk, Alan M. Dunn, Benjamin Livshits, Helen J. Wang, Eyal Ofek

TL;DR

Augmented Reality (AR) applications often require access to sensitive raw sensor data, posing privacy risks. This paper introduces "recognizers," a new OS abstraction that allows AR apps to request higher-level, semantic objects (like a skeleton or a face) instead of raw data. This enables a fine-grained permission system, "privacy goggles" for user visualization, and improved performance through sharing and offloading.

What Problem Does This Paper Address?

Augmented reality (AR) applications, from mobile AR browsers to motion-sensing games, are becoming increasingly sophisticated. These applications rely on sensing the environment through sensors like cameras and microphones to overlay digital content onto the real world. However, current operating systems (OS) offer only coarse-grained access to this sensor data, typically requiring applications to request access to raw streams (e.g., full video feeds). This presents a significant privacy challenge because raw sensor data can contain a wealth of sensitive information, such as users' faces, private surroundings, or even medical details, far beyond what the application actually needs. This "least privilege" principle is difficult to enforce, leading to potential privacy leaks.

Methods

The core innovation is the recognizer abstraction. Instead of raw sensor data, AR applications can request access to "recognized objects" provided by recognizers. A recognizer is an OS component that takes raw sensor input and outputs structured data about recognized entities (e.g., a skeleton's pose, a face's features). This allows for a fine-grained permission system where users grant access to specific recognizer outputs, not raw streams.

To help users understand these permissions, the authors introduce privacy goggles, an "application-eye view" that visualizes the specific information an application will receive. This visualization is crucial for informed consent.

The system also addresses recognizer errors (false positives and negatives) through OS-level error correction techniques like blurring, frame subtraction, and recognizer combination.

Finally, the paper explores performance benefits by leveraging a dataflow graph of recognizers. This enables:

  • Recognizer Sharing: Running computationally intensive recognizers once and sharing their output among multiple applications.
  • On-demand Invocation: Only activating recognizers when needed.
  • Concurrent Execution: Parallelizing independent recognizer computations.
  • Offloading: Moving heavy recognizer computations to a remote server, especially for resource-constrained devices.

Datasets

The paper uses several datasets for evaluation:

  • Kinect-enabled applications: An analysis of 87 shipping AR applications on the Xbox Kinect platform was performed to identify "core recognizers."
  • Well-known vision datasets: Used for evaluating recognizer accuracy, including a Berkeley dataset (objects), an INRIA dataset (talking head), and a dataset of faces turning towards and away from the camera.
  • Survey data: Collected from 462 respondents to evaluate the effectiveness of privacy goggles and user privacy expectations.

Key Insights and Ideas

  • The "Least Privilege" Principle is Achievable for AR: By abstracting sensor data into semantic "recognizers," we can grant applications access to only the specific information they need, significantly enhancing privacy.
  • Users Understand Semantic Information Better: Raw sensor data is overwhelming. Visualizing what an application actually sees through "privacy goggles" is far more effective for user comprehension than traditional permission prompts.
  • A Small Set of Recognizers Covers Most AR Apps: The analysis of 87 applications revealed that a core set of just four recognizers (skeleton tracking, hand position, person texture, and voice commands) is sufficient for nearly 90% of current AR applications, making the abstraction practical.
  • Privacy Enhancements Can Drive Performance: The recognizer abstraction not only improves privacy but also unlocks significant performance gains through sharing, on-demand processing, and offloading. This is a rare win-win scenario.
  • Recognizer Errors are a Real Privacy Concern: False positives from recognizers can leak sensitive information, even if the application only sees a higher-level object. OS-level error correction is necessary.

Key Results

  • A set of four core recognizers covers approximately 89.65% of analyzed shipping Xbox AR applications.
  • No application analyzed required continuous raw RGB access, highlighting the potential for privacy improvements.
  • Surveys showed that 86% of users found raw RGB frames more sensitive than face detector output.
  • Privacy goggles were found to be effective at communicating application capabilities to users, with over 80% correctly identifying capabilities in test scenarios.
  • The prototype system incurred negligible overhead for single applications compared to direct Kinect SDK usage.
  • Recognizer sharing enabled 25 frames per second or higher for up to six concurrent applications, a significant improvement over non-sharing scenarios.
  • Offloading a resource-intensive plane recognizer to a server allowed 4.2 frames per second on a tablet, compared to zero locally. Face recognition offloading improved response time by 34%.

Limitations and Future Work

  • Third-Party Recognizers: The current system assumes trusted recognizers. Supporting third-party recognizers introduces challenges related to permissions, sandboxing, and performance without sacrificing security.
  • Recognizer Visualization: Designing clear and effective visualizations for all types of recognizers (especially non-visual ones like voice commands) remains a challenge.
  • Bystander Privacy: The current system focuses on protecting the primary user's privacy. Addressing bystander privacy in shared AR environments (like Google Glass) is an important future direction.
  • Recognizer Error Correction: While techniques like blurring and frame subtraction were evaluated, further research into more robust and efficient error correction methods is needed.
  • Interrupt-Driven Multiplexer: The current prototype uses a polling-based multiplexer; an interrupt-driven approach could further improve efficiency.

Summary updated

BibTeX
@inproceedings{jana2013enabling,
  author = {Suman Jana and David Molnar and Alexander Moshchuk and Alan M. Dunn and Benjamin Livshits and Helen J. Wang and Eyal Ofek},
  title = {{Enabling Fine-Grained Permissions for Augmented Reality Applications with Recognizers}},
  booktitle = {USENIX Security},
  year = {2013},
}
Figure