SaccadeX: Directed Acyclic Graph-based Semi-Supervised Learning of Continuous Ocular Dynamics from Sparse Neuromorphic Streams

Thivya Kandappu

TL;DR

This paper introduces SaccadeX, a novel graph-based semi-supervised learning framework for continuous eye tracking using neuromorphic event cameras. It addresses the challenges of sparse labels and temporal dynamics in event data by representing events as directed acyclic graphs (DAGs) and employing a teacher-student strategy for label propagation, achieving state-of-the-art results with significantly reduced parameters.

What Problem Does This Paper Address?

Traditional eye-tracking methods often rely on dense, synchronized RGB frames, which struggle to capture the high-speed, millisecond-scale dynamics of eye movements. Neuromorphic event cameras offer a promising alternative due to their sub-microsecond latency and high dynamic range, capturing brightness changes asynchronously. However, existing event-based eye-tracking approaches face several hurdles:

  • Limited Labels: Obtaining dense, synchronized labels for event data is extremely challenging and time-consuming.
  • Sub-optimal Event Accumulation: Simply aggregating events into frames can lead to loss of temporal fidelity and blurred representations.
  • Lack of Fine-Grained Temporal Modeling: Existing methods often fail to effectively capture the subtle, high-frequency temporal relationships within event streams, which are crucial for understanding ocular dynamics like saccades and pupil changes.

This paper aims to bridge this gap by developing a framework that can learn effectively from sparse event data, preserving its native temporal resolution and capturing complex ocular dynamics for applications like gaze tracking, pupil tracking, and emotion recognition.

Methods

SaccadeX introduces several key innovations:

  • Directed Acyclic Graph (DAG) Representation: Instead of traditional frame-based approaches, SaccadeX represents temporally segmented event volumes as DAGs. Each event is a node, and edges connect events based on spatial proximity and temporal order (event $v_i$ to $v_j$ only if $t_i < t_j$). This preserves spatial locality, temporal causality, and avoids frame-based aggregation, maintaining the sensor's native temporal fidelity.
  • Hawkes Process Edge Encoding: The edges in the DAG are encoded using features derived from Hawkes processes. This allows the model to learn how past events influence the likelihood of future events, effectively capturing temporal information propagation.
  • Dual-Branch Spatio-Temporal GNN: The core of the framework is a GNN with two branches:
  • Spatial Branch: Captures morphological information (e.g., eyelid shape) using a Graph Convolutional Network (GCN).
  • Temporal Branch: Encodes event timestamps to capture motion dynamics (e.g., pupil displacement) using an LSTM. These branches are concatenated to form a comprehensive spatio-temporal representation.

  • Semi-Supervised Learning via Teacher-Student Strategy: To overcome label scarcity, SaccadeX employs a teacher-student framework. The "teacher" is the fully trained DAG-based GNN, which generates high-confidence predictions for entire event volumes. The "student" learns from partitioned event volumes, generating "pseudo-labels" for segments. These pseudo-labels are then aligned with the teacher's predictions, allowing the model to propagate sparse supervision across the entire event stream and refine temporal consistency.

  • Curriculum Learning: The framework utilizes curriculum learning to handle the heterogeneity of graphs generated from event data. It starts by training on simpler, more structurally reliable graphs (higher node counts) before progressing to more complex, potentially noisy or incomplete graphs, improving training stability and generalization.
  • Adaptable Architecture: The DAG-based GNN is designed to be modular, allowing for adaptations to different tasks like emotion recognition (replacing regression with classification) and binocular gaze tracking (utilizing latent vectors from both eyes).

Datasets

The paper evaluates SaccadeX on four benchmark datasets:

  • 3ET+ [41]: Used for pupil tracking.
  • EV-Eye [50]: Used for gaze tracking.
  • EBV-Eye [1]: Used for gaze tracking.
  • SEE [48]: Used for emotion recognition.

The paper does not introduce new datasets but focuses on adapting existing ones for event-based analysis and evaluating performance under varying label sparsity.

Key Insights and Ideas

  • DAGs are Key for Event Data: Representing event streams as DAGs is a powerful way to preserve their inherent temporal causality and spatial structure without resorting to frame-based approximations. This is crucial for capturing the fine-grained dynamics of eye movements.
  • Semi-Supervision is Essential: Given the difficulty of obtaining dense labels for event data, a robust semi-supervised approach is critical. The teacher-student strategy effectively leverages limited labels to generate temporally consistent pseudo-labels, significantly improving performance.
  • Curriculum Learning for Robustness: Event data can lead to highly variable graph structures. Curriculum learning provides a structured way to train the model, starting with easier examples and gradually introducing complexity, leading to more stable and generalizable results.
  • Modularity for Versatility: The proposed DAG-based GNN architecture is not limited to a single task. Its modular design allows for straightforward adaptation to related ocular analysis tasks, demonstrating its broad applicability.

Key Results

SaccadeX demonstrates significant improvements across multiple tasks:

  • Parameter Efficiency: Achieves 95.5% parameter reduction compared to existing methods for pupil tracking.
  • Pupil Tracking: 38.75% improvement in accuracy over MambaPupil on the 3ET+ dataset.
  • Gaze Tracking: 68% and 63% reductions in gaze angle error on the EV-Eye and EBV-Eye datasets, respectively, compared to existing SOTA methods.
  • Emotion Recognition: 3.3% improvement in both weighted and unweighted average recall on the SEE dataset.
  • Computational Efficiency: Exhibits significant reductions in FLOPs and inference latency, making it suitable for real-time applications. For instance, it shows an 18-fold latency reduction in gaze tracking.

Limitations and Future Work

  • Reliance on Sparse RGB Labels: While the semi-supervised approach mitigates label sparsity, it still relies on some sparse RGB-derived labels as temporal anchors. Inaccuracies or biases in these initial labels can propagate.
  • Computational Demands: Despite optimizations, processing large-scale event streams remains computationally demanding, especially for embedded systems.

Future work suggested by the authors includes:

  • Exploring self-supervised or contrastive learning methods to further reduce dependence on external labels.
  • Investigating model compression techniques (pruning, quantization) to enhance real-time performance for resource-constrained devices.

Summary updated

Abstract

Continuous eye tracking is critical for applications in human-computer interaction, including biometric authentication, gaze-based systems, and affective-cognitive modeling. Recent interest in neuromorphic event cameras has grown due to their sub-microsecond latency in capturing eye movement dynamics. However, existing event-based eye-tracking methods face challenges such as limited labels, sub-optimal event accumulation, and a lack of frameworks that fail to capture fine-grained temporal relationships within event volumes. To address these, we propose a directed acyclic graph-based semi-supervised approach with a framework that is adaptable across multiple closely related tasks, including gaze tracking, pupil tracking, and eye-based emotion recognition. Our approach enables efficient spatiotemporal learning with 95.5% parameter reduction compared to existing methods, achieving significant performance improvements: 38.75% improvement in pupil tracking accuracy, 68% and 63% reductions in gaze angle error on EV-Eye and EBV-Eye datasets respectively, and 3.3% improvement in emotion recognition across all evaluated dataset.
BibTeX
@inproceedings{kandappu2026saccadex,
  author = {Thivya Kandappu},
  title = {{SaccadeX: Directed Acyclic Graph-based Semi-Supervised Learning of Continuous Ocular Dynamics from Sparse Neuromorphic Streams}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
  doi = {10.1109/wacv61042.2026.00141},
}
Figure