TL;DR
This paper introduces ARETT, an open-source toolkit for Augmented Reality (AR) eye tracking using the Microsoft HoloLens 2. It provides a Unity 3D package for seamless data acquisition and an R package for analysis, addressing the need for robust AR eye tracking research tools. The toolkit's spatial accuracy and precision were evaluated, showing performance comparable to state-of-the-art mobile eye trackers.
What Problem Does This Paper Address?
Augmented Reality (AR) and Virtual Reality (VR) are increasingly incorporating integrated eye trackers into Head-Mounted Displays (HMDs). These trackers offer exciting possibilities for applications like rendering optimization (foveated rendering) and intuitive gaze-based user interaction. Furthermore, understanding visual attention in AR/VR is crucial for cognitive and educational research. While some toolkits exist for VR eye tracking, there's a significant gap in readily available, research-oriented tools specifically for AR scenarios. Existing commercial solutions often lack deep integration with AR development environments, and manufacturer-provided APIs are typically geared towards interaction rather than detailed research data acquisition. This paper aims to bridge this gap by providing a comprehensive, open-source toolkit for AR eye tracking.
Methods
The core of the ARETT toolkit is built upon the Microsoft HoloLens 2's eye tracking capabilities. The toolkit consists of two main parts:
- Unity 3D Package: This package provides a data access layer that interfaces with the HoloLens 2's UWP API to retrieve raw, unsmoothed gaze data. It ensures a stable data rate and high-precision timestamps. The system captures gaze origin, direction, and timestamps, and can optionally record intersections with scene objects (including custom "Areas of Interest" or AOIs) and front-facing camera imagery. A web-based control interface allows experimenters to manage recordings remotely.
- R Package: This package is designed for post-hoc analysis of the collected gaze data. It includes functions for data pre-processing (gap filling, noise reduction) and supports three common offline fixation detection algorithms: I-VT (velocity threshold), I-DT (dispersion threshold), and I-AOI (area of interest-based). It also offers post-processing functions like merging adjacent fixations and discarding short ones.
A key methodological contribution is the comprehensive evaluation of the HoloLens 2's eye tracking accuracy and precision in various conditions. This involved a user study with 21 participants performing fixation tasks while seated and walking, at different distances.
Datasets
The paper does not introduce new datasets for training. Instead, it focuses on collecting and evaluating gaze data from human participants using the HoloLens 2. The evaluation dataset consists of gaze recordings from 21 participants under three different experimental conditions (resting, walking, and a more realistic setting with a stationary target while moving).
Key Insights and Ideas
- Bridging the AR Eye Tracking Gap: The primary insight is the successful development and validation of a robust, open-source toolkit that significantly lowers the barrier for researchers wanting to conduct eye tracking studies in AR.
- HoloLens 2 Performance: The HoloLens 2's integrated eye tracker, when used with ARETT, demonstrates accuracy and precision comparable to established mobile eye trackers, making it a viable option for research.
- Impact of Movement: The study highlights that walking significantly degrades both spatial accuracy and precision compared to resting, a crucial consideration for designing AR experiments.
- Distance Matters: Spatial accuracy is shown to be distance-dependent, with accuracy generally increasing at further distances in the resting condition. This suggests calibration distance plays a role.
- Practical Guidance: The research provides concrete recommendations for minimum target sizes in AR applications based on the measured accuracy and precision, differentiating between stationary and moving users.
Key Results
- Overall Accuracy & Precision: The toolkit and HoloLens 2 achieved an average angular accuracy of 0.83 degrees and a precision of 0.27 degrees during resting conditions (setting I).
- Impact of Walking: When participants were walking (setting II), the average angular accuracy increased to 1.77 degrees, and precision degraded to 1.13 degrees.
- Distance Effects: In resting conditions, accuracy varied from 1.00 degrees (0.5m) to 0.68 degrees (4.0m). In walking conditions, accuracy was higher at closer distances (e.g., 2.52 degrees at 0.5m) and decreased at further distances (e.g., 1.33 degrees at 4.0m).
- Target Size Recommendations: For a 2.0m distance, a minimum target size of 11.10 cm is suggested for resting users, while moving users would require a significantly larger target of 35.42 cm.
- Comparison to Other Trackers: The HoloLens 2's performance in setting I was found to be similar to the Pupil Labs 120 Hz Binocular glasses (0.84° accuracy, 0.16° precision) and better than SMI Eye Tracking Glasses 2 and Tobii Pro Glasses 2 in some metrics.
Limitations and Future Work
- Device Specificity: The current toolkit is limited to the Microsoft HoloLens 2. While the data access layer is designed for modularity, adapting it to other devices will require specific implementation.
- API Limitations: The toolkit relies on the HoloLens 2 API, which provides a joint gaze vector for both eyes. This necessitates intersecting the gaze ray with the environment to determine a gaze point, unlike some binocular trackers that can provide separate rays.
- Experiment Order: The study used a fixed order for experimental settings, which could introduce order effects.
- Limited Interaction Settings: The evaluation focused on specific fixation tasks and may not generalize to all AR interaction scenarios.
- Future Work: The authors plan to:
- Integrate full video recording of the integrated camera.
- Add real-time streaming of video and gaze data.
- Investigate attaching virtual AOIs to real objects for real-time gaze-to-AOI mapping.
- Extend the R package and integrate interfaces for other gaze data processing tools.
- Develop data access layers for other devices.
Summary updated