TL;DR
This paper introduces a novel Gaze-Vergence-Controlled (GVC) see-through vision system for Augmented Reality (AR). By leveraging eye vergence to estimate gaze depth, the system allows users to intuitively control what they see through virtual walls or objects, offering a more natural and efficient interaction than traditional methods like midair clicks or speech commands.
What Problem Does This Paper Address?
Augmented Reality (AR) see-through vision, which allows users to perceive objects behind occluding surfaces, is a fascinating area. While much research has focused on improving the visual realism of this effect, the interaction aspect has been largely overlooked. The authors argue that conventional interaction methods like midair clicks or voice commands are not ideal for controlling see-through vision. This is because the act of wanting to see through something is intrinsically linked to where our eyes are focusing (gaze depth/vergence). Requiring external commands breaks this natural connection, demanding extra cognitive effort and potentially disrupting the user experience. The paper aims to bridge this gap by developing an intuitive and efficient interaction method for AR see-through vision that directly uses eye vergence.
Methods
The core of the proposed system involves two main components: gaze depth estimation and Gaze-Vergence-Controlled (GVC) interaction modes.
- Gaze Depth Estimation: To enable gaze-depth-based control, the researchers first built a custom gaze tracking module by integrating two infrared cameras into a Microsoft HoloLens 2. They then developed and implemented two methods for gaze depth estimation:
- 3D Line-of-sight Intersection (3D LosI): This method reconstructs the 3D gaze point by finding the intersection of the gaze rays from both eyes. They improved upon existing methods by incorporating robust pupil detection and a hardware calibration procedure to account for the kappa angle (the offset between optical and visual axes).
- IPD-based Regression: This approach uses the Inter-Pupillary Distance (IPD) to regress gaze depth. They explored both physical (MIPD) and pixel-based (PIPD) IPD measurements and found that an exponential regression model, combined with RANSAC for outlier rejection, provided accurate results.
- Gaze-Vergence-Controlled (GVC) Modes: Based on the estimated gaze depth, two control modes for see-through vision were proposed:
- Stimulus-Guided (SG) See-through Mode: In this mode, a semi-transparent virtual stimulus is presented at a fixed distance behind the occluding surface. Users trigger see-through vision by focusing on this stimulus, which naturally increases their gaze vergence beyond a threshold.
- Self-Control (SC) See-through Mode: This mode allows users to directly control see-through vision by performing voluntary vergence eye movements without an external stimulus. This is presented as a more novel and appealing interaction.
The system also includes a "Hidden Scene Capture" module that uses a camera behind the occluding wall, registers its pose to the HoloLens, extracts the Region of Interest (ROI) based on the user's gaze, and performs perspective transformation to render the hidden scene realistically within the user's view.
Datasets
The paper does not explicitly mention using pre-existing public datasets for training or evaluation. Instead, they describe custom experimental setups and participant recruitment for their evaluations. The "calibration scene" and "test scenario" described in the paper were created by the authors for their specific experiments.
Key Insights and Ideas
- Natural Interaction: The fundamental insight is that controlling see-through vision should be as natural as how we naturally focus our eyes. Linking interaction directly to gaze vergence taps into this innate human capability.
- Addressing Midas Touch: By using vergence eye movement, the system inherently avoids the "Midas Touch" problem (unintentional selections from natural eye movements) that plagues gaze-dwelling techniques. Vergence is a more deliberate eye movement.
- Dual Gaze Depth Estimation: The paper highlights that different gaze depth estimation methods have varying strengths. 3D LosI is more accurate at closer ranges, while IPD-based regression performs better at longer distances, suggesting a piecewise approach can optimize performance.
- User Preference for Gaze: The experimental results strongly suggest that users find gaze-based control more intuitive and enjoyable than traditional methods, especially the SCGaze mode.
Key Results
- Gaze Depth Estimation Accuracy: The proposed 3D LosI method achieved an error of 0.41 ± 0.34 m in the (0.5, 2] m range, while PIPD outperformed at longer distances (1.14 ± 0.49 m in the (2, 6] m range). Their combined piecewise function yielded an error of 0.57 ± 0.44 m for daily indoor interaction (0.5, 3] m.
- Efficiency: In objective evaluations, GVC techniques (SGGaze and SCGaze) were significantly faster than midair click and speech commands. For example, SGGaze was faster than Click and Speech (p < 0.001).
- Usability: User feedback indicated that GVC techniques were more intuitive and less frustrating. Users reported less mental and physical fatigue compared to click-based interactions.
- User Preference: The SCGaze mode was the most preferred by users (60% ranked it first), followed by SGGaze (30%). Users found the gaze-based control "amazing" and a "novel experience."
- Accuracy (Success Rate): The number of successful operations was not significantly different across modalities, indicating that while GVC might have occasional false triggers, users could still complete tasks effectively.
Limitations and Future Work
- Distance Limitations: The paper notes that for distances exceeding 3 meters, discriminating vergence differences becomes difficult. For such long-distance interactions, they suggest modality-independent methods like speech might be more suitable.
- Stimulus Distraction: While SGGaze was efficient, some users reported the stimulus could be distracting. The authors plan to reduce stimulus transparency in future work.
- Shared See-Through Vision: A future direction is to enable shared see-through vision experiences among multiple users, where each user controls their view via gaze vergence, potentially using cameras from other users' devices.
- Error Tolerance: The authors recommend using error-tolerant designs, such as distance thresholds and smoothing filters, to improve the robustness of GVC techniques.
- Fixed Window Depth: They suggest fixing the see-through vision window depth to prevent drift and stabilize gaze, thereby reducing visual fatigue.
Summary updated