Sub-Millisecond Event-Based Eye Tracking on a Resource-Constrained Microcontroller

Marco Giordano, Pietro Bonazzi, Luca Benini, Michele Magno

Abstract

Eye tracking is a critical component for the next generation of wearable Augmented and Virtual Reality (AR/VR) devices, yet traditional frame-based systems often struggle with high latency and significant power consumption. This paper presents a novel, end-to-end eye-tracking pipeline that leverages event-based cameras and a hardware-aware, quantized Convolutional Neural Network (CNN) running on a resource-constrained microcontroller. By processing data in the event domain and utilizing specialized hardware acceleration, the system achieves a total end-to-end latency of 385 $\mu$s and milliwatt-scale power consumption, enabling high-frequency tracking suitable for mobile, battery-powered applications.

Introduction

Modern AR/VR headsets require precise, low-latency eye tracking to support features like foveated rendering and intuitive user interfaces. However, conventional systems relying on frame-based cameras face three primary challenges: high latency due to fixed frame rates, high power consumption from processing full image frames, and sensitivity to lighting conditions.

Event-based cameras, or Dynamic Vision Sensors (DVS), offer a compelling alternative by reporting pixel-level intensity changes asynchronously. While these sensors provide high temporal resolution and high dynamic range, processing their high-speed data streams typically requires significant computational resources, such as GPUs or high-end FPGAs. This research bridges the gap by demonstrating a complete eye-tracking system that fits within the power and computational constraints of a standard microcontroller (MCU).

System Architecture and Methodology

The proposed system utilizes the STM32N6 microcontroller, featuring the Neural-ART Accelerator (NPU), paired with the DVXplorer Micro event camera. The pipeline is designed for maximum efficiency through several key innovations:

Event Slicing and Representation

Traditional event processing often accumulates events over a fixed time window, leading to variable data density depending on the speed of eye movement. To ensure a consistent signal for the neural network, the authors implement event-count slicing. The system accumulates exactly 1,000 events into a 2-channel 64x64 frame. This approach ensures that the CNN always receives a stable amount of visual information, regardless of whether the user is performing a rapid saccade or a slow fixation.

Compact CNN Design

The core of the tracking system is a lightweight CNN architecture consisting of six convolutional blocks followed by a fully connected head. The model is specifically optimized for low-resolution (64x64) inputs to minimize the number of Multiply-Accumulate (MAC) operations required per inference.

Hardware-Aware Optimization

To achieve sub-millisecond performance on an MCU, the model underwent static post-training quantization. By converting 32-bit floating-point weights into 8-bit integers (INT8), the researchers drastically reduced the memory footprint and enabled the use of the STM32N6’s dedicated NPU. This hardware-software co-design allows the system to execute the entire pipeline—from event accumulation to pupil center prediction—entirely on-device.

Experimental Results

The system was evaluated using the Ini-30 dataset, a benchmark for event-based eye tracking. The results demonstrate that high-speed tracking does not require high-power hardware.

Latency and Throughput

The total end-to-end processing time was measured at 385 $\mu$s. This latency is distributed as follows:

  • Preprocessing: 119 $\mu$s
  • Data Loading: 62 $\mu$s
  • NPU Inference: 204 $\mu$s

The Neural-ART Accelerator achieved a throughput of 52 MAC operations per cycle, showcasing the efficiency of the quantized architecture.

Accuracy and Power Consumption

Despite its compact size, the system achieved a median pupil prediction error of 5.76 pixels on the Ini-30 dataset, remaining competitive with state-of-the-art models that require significantly more computational power.

In terms of energy efficiency, each inference consumes only 155 $\mu$J. When operating at a standard 120Hz tracking rate, the total system power consumption is approximately 33 mW. This efficiency allows for a projected battery life of approximately 16 hours on a typical smart glasses battery, as the system remains in an idle state for roughly 95% of the time between updates.

Discussion and Future Work

A key insight of this work is the "Information Density" advantage: by slicing the event stream based on event count rather than time, the system becomes more robust across different users and movement speeds. Furthermore, the ability to exploit idle time between inferences is the primary driver for the system's extreme energy efficiency.

Future work will focus on tighter integration between the DVS sensor and the MCU's memory bus to further reduce data transfer overhead. Additionally, the researchers plan to test the system in more diverse "in-the-wild" scenarios to ensure robustness against varying lighting conditions and user physiologies beyond the standard benchmarks.

Conclusion

This research demonstrates that sub-millisecond, high-accuracy eye tracking is achievable on resource-constrained microcontrollers. By combining event-based sensing with hardware-aware neural network optimization, the authors have provided a blueprint for the next generation of ultra-low-power wearable devices.

Summary updated

BibTeX
@inproceedings{giordano2025sub,
  author = {Marco Giordano and Pietro Bonazzi and Luca Benini and Michele Magno},
  title = {{Sub-Millisecond Event-Based Eye Tracking on a Resource-Constrained Microcontroller}},
  booktitle = {IWASI},
  year = {2025},
  doi = {10.1109/iwasi66786.2025.11121976},
}
Figure