Event-Based Kilohertz Eye Tracking Using Coded Differential Lighting

Timo Stoffregen, Hossein Daraei, Clare Robinson, Alexander FixWACV 2022

TL;DR

This paper introduces a novel event-based eye tracking system that achieves kilohertz sampling rates with low power consumption, making it ideal for AR/VR applications. The core innovation is "Coded Differential Lighting," a new illumination scheme that enhances corneal glints while suppressing background noise, enabling robust and precise eye tracking even during rapid eye movements.

What Problem Does This Paper Address?

Accurate and low-latency eye tracking is crucial for immersive technologies like Augmented and Virtual Reality (AR/VR). Current systems often struggle with power consumption and real-time responsiveness, especially during rapid eye movements (saccades). Traditional cameras, with their fixed frame rates and susceptibility to motion blur, are not ideal for these demanding applications. While event cameras offer advantages like low power and microsecond latency, effectively leveraging them for precise tasks like corneal glint detection, a key component of gaze estimation, has been challenging. Specifically, the paper addresses the difficulty of reliably detecting corneal glints at high frequencies without being overwhelmed by spurious events from other parts of the eye or scene, and the problem of unambiguously associating detected glints with their corresponding light sources.

Methods

The paper proposes an "Event-Based Kilohertz Eye Tracking" system that combines several novel techniques:

  • Coded Differential Lighting: This is the central innovation. Instead of simply flashing LEDs, the authors use pairs of LEDs that toggle on and off in a compensatory manner. This keeps the overall scene illumination relatively constant, minimizing spurious events from diffuse surfaces like skin and iris. However, specular reflections (glints) from the cornea are still generated and are enhanced by this differential lighting.
  • Binary Glint Encoding: To overcome the bandwidth limitations of event cameras and support a large number of light sources, the paper introduces a binary encoding scheme. Each light source (LED) is pulsed with a unique binary sequence (e.g., short pulse for '0', long pulse for '1'). This allows for arbitrary numbers of light sources to be distinguished using only two base frequencies, which are then filtered.
  • Frequency Filtering (FF): Events are filtered based on the expected frequencies of the binary encoded light pulses. This process effectively isolates events originating from the glints and suppresses noise from other sources.
  • Binary Glint (BG) Tracking: A tracker object is maintained for each potential glint. These trackers operate in different states (uninitialized, semi-initialized, fully initialized) and use the frequency-filtered "glint-maps" to update their position and decode the binary sequence, thereby identifying the glint's source.
  • Corneal Sphere Regression: Once glints are detected and their sources identified, the system models the cornea as an ideal sphere. By optimizing the reprojection error of known light source positions onto the image plane, the system estimates the corneal sphere's dimensions and location, which is fundamental for gaze estimation.

Datasets

The paper does not explicitly mention training datasets. For evaluation, they used:

  • Model Eyes: Experiments were conducted using a model eye on a rotation stage to measure accuracy against angular velocity.
  • Real Eye Motions: Sequences of saccadic and smooth-pursuit motions were recorded from a human participant. Ground truth was established by hand-labeling glint centroids from event frames.

Key Insights and Ideas

  • Differential Lighting is Key for Event Cameras: The core insight is that by carefully controlling illumination to maintain a near-constant overall scene brightness while introducing controlled specular reflections, event cameras can be made to detect specific specular targets (corneal glints) with high signal-to-noise ratio, while ignoring diffuse reflections.
  • Binary Encoding Solves Scalability: The binary encoding scheme is a clever solution to the problem of identifying multiple light sources with limited bandwidth, allowing for a virtually unlimited number of distinct glint sources.
  • Event Cameras Excel at High-Speed Dynamics: The paper demonstrates that event cameras, when paired with appropriate lighting and processing, can track dynamic phenomena like rapid eye movements at kilohertz rates, far exceeding the capabilities of conventional cameras for such tasks.
  • Robustness to Noise: The frequency filtering approach inherently rejects noise outside the encoded frequencies, making the system robust to flickering lights, ambient illumination changes, and other spurious events common in real-world AR/VR environments.

Key Results

  • Sub-pixel Accuracy: The system achieves sub-pixel accuracy in glint detection, with a mean L2 error of 0.342 pixels for saccadic motions and 0.497 pixels for smooth-pursuit motions on real eyes.
  • High Angular Velocity Performance: The method can detect corneal glints with below 0.5 pixel error even at angular velocities up to 1000°/s, significantly exceeding the typical range of human ocular motion (around 500°/s).
  • Kilohertz Sampling Rate: The system operates at a 1 kHz sampling rate, enabling real-time updates crucial for AR/VR applications.
  • Low Power Consumption: The overall system power consumption is estimated to be around 35 mW at typical human ocular motion speeds.
  • Robustness: The method demonstrates robustness to background events and sensor noise, maintaining sub-pixel accuracy even when subjected to significant external light sources.

Limitations and Future Work

  • Hardware Limitations: The paper notes that the maximum achievable sampling rate is limited by the event camera's hardware capabilities and biases, and that higher frequencies can lead to reduced detection accuracy. They were unable to find biases that would permit sampling rates faster than 1 kHz.
  • Calibration Dependency: Like most eye-tracking systems, this method relies on device calibration to know the fixed positions of the LEDs.
  • Future Work: The authors suggest that the differential illumination scheme could be applied more generally to specular object imaging with event cameras. They also express hope that their work will inspire further use of event sensors in actual eye-tracking solutions.

Summary updated

BibTeX
@inproceedings{stoffregen2022event,
  author = {Timo Stoffregen and Hossein Daraei and Clare Robinson and Alexander Fix},
  title = {{Event-Based Kilohertz Eye Tracking Using Coded Differential Lighting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure