Introduction
In the field of human-robot collaboration, a fundamental challenge is enabling robots to understand which objects a human is interacting with in real-time. Traditional computer vision (CV) solutions, such as Faster R-CNN or YOLO, are the standard for object detection. However, these models are computationally expensive, require high-end GPUs, and often struggle to generalize to objects outside their specific training sets.
This research explores a radical alternative: detecting objects and estimating their bounding boxes using only human gaze data. By bypassing image-based processing entirely, the researchers demonstrate that a human's eye movements inherently encode the location and presence of objects in the environment.
Methodology: From Gaze to Heatmaps
The core of this approach is the transformation of raw gaze trajectories into spatial distributions. Rather than treating gaze as a simple sequence of coordinates, the method groups gaze points into temporal windows (ranging from 100ms to 500ms) to capture the dynamics of visual attention.
2D and 3D Heatmap Generation
These gaze points are mapped onto a grid to create a "heatmap." The study compared two primary formats:
- 2D Heatmaps: Standard spatial mapping of gaze coordinates.
- 3D Heatmaps: These incorporate depth information (distance from the eye), calculated via specialized eye-tracking software. This additional dimension allows the model to distinguish between objects at different depths in the user's field of view.
Lightweight Machine Learning
The flattened heatmaps serve as inputs for lightweight machine learning models. Because the input is a small grid (e.g., 10x10 to 50x50 pixels) rather than a high-resolution image, the computational requirements are drastically reduced. The researchers utilized:
- Classification: K-Nearest Neighbors (KNN), Bagged Trees, and Gaussian SVMs to determine if a gaze window corresponds to an object.
- Regression: Gaussian Process Regression to predict the bounding box parameters (x, y, width, and height).
Data Collection "In the Wild"
To validate this approach in realistic scenarios, the authors created a new dataset using Pupil Invisible eye-tracking glasses. Five participants wore the glasses while navigating diverse real-world environments, including university offices, corridors, streets, and parking lots. The resulting dataset comprises approximately one hour of recording (102,620 frames), with nearly 28,000 frames containing labeled objects like fire extinguishers, signs, and bicycles.
Key Results and Efficiency
The experimental results highlight the viability of gaze-based detection as a low-power alternative to traditional CV:
- High Accuracy: The best classification performance reached 92% accuracy using 3D heatmaps with a KNN classifier. The inclusion of depth information significantly improved results over 2D-only models.
- Extreme Efficiency: A Gaussian SVM on a 2D heatmap can perform 1,000 predictions in less than one second on a standard CPU.
- Minimal Memory Footprint: While a model like Faster R-CNN typically requires ~1.7 GB of RAM for a single prediction, the gaze-based Gaussian SVM requires less than 1 MB. This represents a reduction in memory usage of over 860x.
- Spatial Prediction: The models were highly successful at predicting the position (X, Y) of objects. However, predicting the exact size (Width, Height) proved more difficult, as humans often fixate on specific features rather than scanning the entire perimeter of an object.
Conclusion and Future Work
This research proves that human gaze is a powerful proxy for object geometry. By treating gaze trajectories as a signal for object detection, robots can identify points of interest with negligible computational overhead. While currently a proof of concept, future work will focus on integrating this gaze-based trigger into live robotic systems, allowing them to use gaze to "wake up" more intensive image-based sensors only when a human interest is detected. This hybrid approach could lead to significantly more energy-efficient and responsive autonomous systems.
Summary updated