EyeGraph: Modularity-aware Spatio Temporal Graph Clustering for Continuous Event-based Eye Tracking

Nuwan Sriyantha Bandara, Thivya Kandappu, Argha Sen, Ila Gokarn, Archan MisraNeurIPS 2024 · OpenReview

TL;DR

EyeGraph introduces a novel unsupervised, graph-based framework and a large-scale "in-the-wild" dataset for high-frequency eye tracking using neuromorphic (event-based) cameras. By treating asynchronous event streams as dynamic, modularity-aware graphs, the researchers achieve pupil-tracking accuracy that rivals supervised models—all without the need for dense, labor-intensive manual labeling.

The Challenge of High-Speed Eye Tracking

Traditional eye tracking relies on standard RGB cameras, which are limited by fixed frame rates (usually 30–60 Hz) and are prone to motion blur during rapid eye movements like saccades. While neuromorphic "event cameras" solve this by recording per-pixel intensity changes at microsecond resolution, they introduce new challenges:

  1. Label Sparsity: Ground truth gaze data is usually captured at much lower frequencies than the event stream, making supervised learning exceptionally difficult.
  2. Representation Mismatch: Most researchers "collapse" events into 2D frames to use standard computer vision tools, which throws away the precise temporal benefits of the sensor.
  3. Lack of Real-World Data: Existing event-based eye datasets are mostly recorded in controlled lab settings with fixed head positions, failing to account for the "in-the-wild" noise of human movement and changing light.

Methods: A Shift to Dynamic Graphs

The authors propose a fundamental shift from "frame-based" thinking to Dynamic Graph Construction, treating the continuous stream of events as a topological structure.

The EyeGraph pipeline, illustrating the transformation of raw event data into a 3D spatio-temporal graph, followed by clustering and tube-model RANSAC for continuous pupil tracking.
  • Spatio-Temporal Graphs: Instead of binning events into a flat picture, they treat each event as a node in a 3D graph defined by spatial coordinates and time ($x, y, t$).
  • Adaptive Edge Construction: To decide which events "belong" together, the system uses a Gaussian Mixture Model (GMM) to dynamically set a thresholding radius. This ensures the graph structure adapts seamlessly to the varying speeds of eye movement.
  • Hawkes Process Attribution: They employ a Hawkes process-based mechanism for edge features, weighting events based on their historical influence to capture the "momentum" of the eye.
  • Unsupervised Topological Clustering: The core of the model is a Variational Graph Autoencoder (VGAE) with a unique twist: the loss function is designed to maximize modularity. In graph theory, modularity measures the strength of division of a network into clusters. This allows the model to naturally "discover" distinct anatomical parts of the eye (pupil, iris, eyelids) because they move with different profiles.
  • Tube-Model RANSAC: Once the pupil cluster is identified, a custom RANSAC (Random Sample Consensus) method fits a "tube" through the spatio-temporal event cloud to estimate the pupil's center coordinates continuously.

The EyeGraph Dataset

To validate their approach and push the field forward, the paper introduces EyeGraph, a significant new benchmark dataset designed for real-world robustness.

Data collection setups for the EyeGraph dataset, showing various 'in-the-wild' conditions, including different head-mounted devices and participant postures.
  • Scale & Diversity: Features 40 participants from diverse ethnic backgrounds.
  • In-the-Wild Conditions: Includes scenarios with unrestricted head and body movement, as well as varying ambient light (ranging from a bright 348 Lux down to a dim 24 Lux).
  • Multi-modal Capture: Recorded using a wearable DAVIS346 event camera with synchronized RGB reference data for comprehensive evaluation.

For broader evaluation, the researchers also benchmarked their method against existing datasets like EBV-Eye and 3ET+.

Key Insights

  • "Pixels that fire together wire together": Inspired by Hebbian learning, the authors treat eye-tracking as a topology discovery task. If events are spatially close and temporally contiguous, they likely belong to the same anatomical structure.
  • Modularity as a Proxy for Labels: By maximizing modularity, the system separates the "tunnel-like" movement of the pupil from the "shutter-like" movement of the eyelids without ever being explicitly trained on what a pupil looks like.
  • Robustness to Noise: Because the graph approach focuses on the relationships between events rather than absolute pixel intensity, it is remarkably resilient to the "salt-and-pepper" noise and motion artifacts that typically plague event cameras in low light.

Results and Performance

  • State-of-the-Art Accuracy: On the 3ET+ dataset, EyeGraph achieved a p10 accuracy of 91.45%, significantly outperforming traditional unsupervised methods like k-means (64.9%) and even beating several supervised baselines.
  • In-the-Wild Reliability: On their proprietary EyeGraph dataset, the method maintained a highly robust 93.67% accuracy even under poor lighting and severe motion artifacts.
  • Scalable Efficiency: By avoiding the onerous task of dense manual labeling, EyeGraph provides a scalable pathway to train high-frequency trackers for AR/VR applications where user diversity and lighting conditions vary wildly.

Limitations and Future Work

While EyeGraph represents a major leap in event-based tracking, the authors note a few limitations. The current study tracks only one eye (monocular focus); future iterations will need binocular tracking for depth perception and medical diagnostics. Additionally, the current data relies on a moving dot stimulus. The team plans to release future datasets featuring more natural stimuli, such as reading or interacting with physical objects. Finally, the next major milestone will be deploying this efficient algorithm directly onto edge devices (like AR glasses) to fully leverage the ultra-low latency of neuromorphic sensors in real-time.

Summary updated

Abstract

Continuous tracking of eye movement dynamics plays a significant role in developing a broad spectrum of human-centered applications, such as cognitive skills (visual attention and working memory) modeling, human-machine interaction, biometric user authentication, and foveated rendering. Recently neuromorphic cameras have garnered significant interest in the eye-tracking research community, owing to their sub-microsecond latency in capturing intensity changes resulting from eye movements. Nevertheless, the existing approaches for event-based eye tracking suffer from several limitations: dependence on RGB frames, label sparsity, and training on datasets collected in controlled lab environments that do not adequately reflect real-world scenarios. To address these limitations, in this paper, we propose a dynamic graph-based approach that uses a neuromorphic event stream captured by Dynamic Vision Sensors (DVS) for high-fidelity tracking of pupillary movement. More specifically, first, we present EyeGraph, a large-scale multi-modal near-eye tracking dataset collected using a wearable event camera attached to a head-mounted device from 40 participants -- the dataset was curated while mimicking in-the-wild settings, accounting for varying mobility and ambient lighting conditions. Subsequently, to address the issue of label sparsity, we adopt an unsupervised topology-aware approach as a benchmark. To be specific, (a) we first construct a dynamic graph using Gaussian Mixture Models (GMM), resulting in a uniform and detailed representation of eye morphology features, facilitating accurate modeling of pupil and iris. Then (b) apply a novel topologically guided modularity-aware graph clustering approach to precisely track the movement of the pupil and address the label sparsity in event-based eye tracking. We show that our unsupervised approach has comparable performance against the supervised approaches while consistently outperforming the conventional clustering approaches.
BibTeX
@inproceedings{bandara2024eyegraph,
  author = {Nuwan Sriyantha Bandara and Thivya Kandappu and Argha Sen and Ila Gokarn and Archan Misra},
  title = {{EyeGraph: Modularity-aware Spatio Temporal Graph Clustering for Continuous Event-based Eye Tracking}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=YxuuzyplFZ},
}
Figure