Dynamic X-Ray Vision in Mixed Reality

Hung-Jui Guo, Jonathan Z. Bakdash, Laura R. Marusich, Balakrishnan PrabhakaranVRST 2022 · DOI · arXiv · OpenReview

TL;DR

This paper introduces a dynamic "X-ray vision" window for Mixed Reality (MR) that allows users to see through physical occlusions in real-time. By integrating scene understanding, eye tracking, and dynamic rendering on a HoloLens 2 device, it aims to provide a more intuitive and natural way to perceive occluded environments, moving beyond static or image-based approaches.

What Problem Does This Paper Address?

In real-world environments, physical objects like walls and furniture often block our view of what's behind them. While tools like minimaps can provide some spatial information, they don't offer direct visual perception of occluded areas. This limitation can be critical in scenarios requiring immediate situational awareness, such as law enforcement or military operations. Existing "X-ray vision" techniques in Augmented and Mixed Reality often rely on static images or fixed windows, which don't adapt to the user's dynamic movement and perspective, hindering natural interaction and depth perception. This paper addresses the need for a more fluid and responsive X-ray vision system that mimics how we naturally perceive the world through a window.

Methods

The core contribution is a dynamic X-ray vision window implemented on the Microsoft HoloLens 2 MR headset. The system leverages several key components:

  • Scene Understanding: The HoloLens 2's built-in scene understanding capabilities are used to map the physical environment. This allows the system to identify surfaces and occlusions.
  • Dynamic Rendering: Instead of a static overlay, the X-ray vision window is rendered dynamically. Its position and transparency are updated in real-time based on the user's current physical location and movement.
  • Eye-Gaze Integration: The system uses the HoloLens 2's eye-tracking feature to determine the user's gaze direction. This information is used to dynamically position and orient the X-ray vision window, making it appear as if the user is looking through a virtual aperture that follows their gaze.
  • Clipping Primitive: The Mixed Reality Toolkit (MRTK) is used to implement a clipping primitive. This feature allows the portion of the virtual content within the X-ray window to be rendered, while also making the contact area with the occluding real-world surface partially transparent, enhancing the illusion of seeing through.

This approach moves beyond previous methods that often used flat images aligned with surfaces, aiming to provide richer depth cues like motion parallax and binocular disparity.

Datasets

Not discussed in the paper.

Key Insights and Ideas

  • Mimicking Natural Perception: The dynamic nature of the window, driven by user position and gaze, is crucial for creating a more natural and intuitive experience, akin to looking through a real window.
  • Beyond Static Overlays: The paper highlights the limitations of static X-ray vision approaches, emphasizing the need for real-time adaptation to user movement for effective spatial understanding.
  • Gaze as an Interaction Cue: Utilizing eye-gaze to control the X-ray window's focus and position is a novel way to integrate user intent directly into the visualization.
  • Potential for Critical Applications: The authors point to significant potential in fields like law enforcement and military for enhanced situational awareness, where knowing what lies beyond line-of-sight is paramount.

Key Results

The paper describes the development and integration of the dynamic X-ray vision window. While specific quantitative results from user studies are not presented in this excerpt, the authors detail the system's architecture and its potential benefits. The key outcome is a functional prototype on the HoloLens 2 that demonstrates real-time rendering of occluded content based on user dynamics and gaze.

Limitations and Future Work

The authors acknowledge that while the system is a step forward, the usability and human performance with this dynamic X-ray vision window still need to be rigorously evaluated. They anticipate that users might find interacting with such a system unfamiliar, uncomfortable, or even distracting.

Future work will focus on:

  • User Study and Evaluation: Conducting a user study, as proposed in Section 2.1, to assess how the dynamic X-ray vision window impacts participants' depth perception, occlusion understanding, and task performance, particularly with physical movement involved.
  • Improving Realism: Enhancing the system's realism by attaching textures to virtual surfaces within the X-ray window.
  • Exploring Game-like Scenarios: Planning to build a game-like user study to explore the benefits of the dynamic X-ray vision window compared to other X-ray vision forms or advanced spatial knowledge techniques, measuring performance through movement direction and reaction time.

Summary updated

Abstract

X-ray vision, a technique that allows users to see through walls and other obstacles, is a popular technique for Augmented Reality (AR) and Mixed Reality (MR). In this paper, we demonstrate a dynamic X-ray vision window that is rendered in real-time based on the user's current position and changes with movement in the physical environment. Moreover, the location and transparency of the window are also dynamically rendered based on the user's eye gaze. We build this X-ray vision window for a current state-of-the-art MR Head-Mounted Device (HMD) -- HoloLens 2 by integrating several different features: scene understanding, eye tracking, and clipping primitive.
BibTeX
@inproceedings{guo2022dynamic,
  author = {Hung-Jui Guo and Jonathan Z. Bakdash and Laura R. Marusich and Balakrishnan Prabhakaran},
  title = {{Dynamic X-Ray Vision in Mixed Reality}},
  booktitle = {VRST},
  year = {2022},
  doi = {10.1145/3562939.3565675},
}
Figure