TL;DR
EyeGraph introduces a novel unsupervised, graph-based framework and a large-scale "in-the-wild" dataset for high-frequency eye tracking using neuromorphic (event-based) cameras. By treating asynchronous event streams as dynamic, modularity-aware graphs, the researchers achieve pupil-tracking accuracy that rivals supervised models—all without the need for dense, labor-intensive manual labeling.
The Challenge of High-Speed Eye Tracking
Traditional eye tracking relies on standard RGB cameras, which are limited by fixed frame rates (usually 30–60 Hz) and are prone to motion blur during rapid eye movements like saccades. While neuromorphic "event cameras" solve this by recording per-pixel intensity changes at microsecond resolution, they introduce new challenges:
- Label Sparsity: Ground truth gaze data is usually captured at much lower frequencies than the event stream, making supervised learning exceptionally difficult.
- Representation Mismatch: Most researchers "collapse" events into 2D frames to use standard computer vision tools, which throws away the precise temporal benefits of the sensor.
- Lack of Real-World Data: Existing event-based eye datasets are mostly recorded in controlled lab settings with fixed head positions, failing to account for the "in-the-wild" noise of human movement and changing light.
Methods: A Shift to Dynamic Graphs
The authors propose a fundamental shift from "frame-based" thinking to Dynamic Graph Construction, treating the continuous stream of events as a topological structure.
- Spatio-Temporal Graphs: Instead of binning events into a flat picture, they treat each event as a node in a 3D graph defined by spatial coordinates and time ($x, y, t$).
- Adaptive Edge Construction: To decide which events "belong" together, the system uses a Gaussian Mixture Model (GMM) to dynamically set a thresholding radius. This ensures the graph structure adapts seamlessly to the varying speeds of eye movement.
- Hawkes Process Attribution: They employ a Hawkes process-based mechanism for edge features, weighting events based on their historical influence to capture the "momentum" of the eye.
- Unsupervised Topological Clustering: The core of the model is a Variational Graph Autoencoder (VGAE) with a unique twist: the loss function is designed to maximize modularity. In graph theory, modularity measures the strength of division of a network into clusters. This allows the model to naturally "discover" distinct anatomical parts of the eye (pupil, iris, eyelids) because they move with different profiles.
- Tube-Model RANSAC: Once the pupil cluster is identified, a custom RANSAC (Random Sample Consensus) method fits a "tube" through the spatio-temporal event cloud to estimate the pupil's center coordinates continuously.
The EyeGraph Dataset
To validate their approach and push the field forward, the paper introduces EyeGraph, a significant new benchmark dataset designed for real-world robustness.
- Scale & Diversity: Features 40 participants from diverse ethnic backgrounds.
- In-the-Wild Conditions: Includes scenarios with unrestricted head and body movement, as well as varying ambient light (ranging from a bright 348 Lux down to a dim 24 Lux).
- Multi-modal Capture: Recorded using a wearable DAVIS346 event camera with synchronized RGB reference data for comprehensive evaluation.
For broader evaluation, the researchers also benchmarked their method against existing datasets like EBV-Eye and 3ET+.
Key Insights
- "Pixels that fire together wire together": Inspired by Hebbian learning, the authors treat eye-tracking as a topology discovery task. If events are spatially close and temporally contiguous, they likely belong to the same anatomical structure.
- Modularity as a Proxy for Labels: By maximizing modularity, the system separates the "tunnel-like" movement of the pupil from the "shutter-like" movement of the eyelids without ever being explicitly trained on what a pupil looks like.
- Robustness to Noise: Because the graph approach focuses on the relationships between events rather than absolute pixel intensity, it is remarkably resilient to the "salt-and-pepper" noise and motion artifacts that typically plague event cameras in low light.
Results and Performance
- State-of-the-Art Accuracy: On the 3ET+ dataset, EyeGraph achieved a p10 accuracy of 91.45%, significantly outperforming traditional unsupervised methods like k-means (64.9%) and even beating several supervised baselines.
- In-the-Wild Reliability: On their proprietary EyeGraph dataset, the method maintained a highly robust 93.67% accuracy even under poor lighting and severe motion artifacts.
- Scalable Efficiency: By avoiding the onerous task of dense manual labeling, EyeGraph provides a scalable pathway to train high-frequency trackers for AR/VR applications where user diversity and lighting conditions vary wildly.
Limitations and Future Work
While EyeGraph represents a major leap in event-based tracking, the authors note a few limitations. The current study tracks only one eye (monocular focus); future iterations will need binocular tracking for depth perception and medical diagnostics. Additionally, the current data relies on a moving dot stimulus. The team plans to release future datasets featuring more natural stimuli, such as reading or interacting with physical objects. Finally, the next major milestone will be deploying this efficient algorithm directly onto edge devices (like AR glasses) to fully leverage the ultra-low latency of neuromorphic sensors in real-time.
Summary updated