BlissCam: Boosting Eye Tracking Efficiency with Learned In-Sensor Sparse Sampling

Yu Feng, Tianrui Ma, Yuhao Zhu, Xuan Zhang

Abstract

Eye tracking is a critical component for modern Augmented and Virtual Reality (AR/VR) systems, enabling features such as foveated rendering and gaze-based interaction. However, existing systems are constrained by significant power consumption and latency, often exceeding 2W and 15ms respectively. BlissCam addresses these bottlenecks by introducing "in-sensor sparse sampling." By integrating motion detection and random sampling directly into the image sensor hardware, BlissCam discards 95% of pixel data at the source. This approach achieves an 8.2× reduction in energy consumption and a 1.4× improvement in end-to-end latency while maintaining high tracking accuracy (gaze error < 1.0°).

Introduction: The Power and Latency Wall

The "power and latency wall" in current AR/VR eye trackers stems from a frontend bottleneck: the image sensor. Standard sensors read out every pixel, forcing the system to move and process massive amounts of redundant data. While many researchers focus on optimizing backend algorithms, BlissCam co-designs the sensor hardware and tracking algorithms to "thin out" data during capture.

Latency and power bottlenecks in high-resolution eye tracking.

Methods: In-Sensor and Off-Sensor Co-Design

The BlissCam pipeline is divided into in-sensor and off-sensor components to maximize efficiency without sacrificing robustness.

1. In-Sensor Sparse Sampling

Using a modified Digital Pixel Sensor (DPS) architecture, BlissCam performs three key steps directly on the chip:

  • Eventification: The sensor compares the current frame to the previous one. Since the camera is fixed relative to the head, the background is stationary, allowing the sensor to focus only on moving parts (the eye).
  • ROI Prediction: A tiny, in-sensor Neural Processing Unit (NPU) uses these "events" to predict a Region-of-Interest (ROI) box around the eye.
  • Random Sampling: Within that ROI, the sensor only quantizes and reads out a small fraction (~5%) of pixels chosen at random.

Hardware efficiency is achieved by time-multiplexing existing Single-Slope ADC components and per-pixel SRAM, resulting in only a 5.8% increase in silicon area.

2. Off-Sensor Robust Segmentation

The host processor receives sparse pixel data resembling a "starry night." BlissCam utilizes a Vision Transformer (ViT) for segmentation. Unlike traditional CNNs, which rely on local spatial continuity, the ViT’s attention mechanism effectively relates distant pixels, making it naturally robust to high levels of data sparsity.

BlissCam pipeline overview and tracking latency components.

Key Results

Evaluation on the OpenEDS dataset demonstrates significant improvements over traditional pipelines:

  • 95% Pixel Volume Reduction: Achieving a 20× compression rate at the source.
  • 8.2× Energy Reduction: Primarily due to reduced activity in the power-hungry readout chain and MIPI interfaces.
  • 1.4× Latency Reduction: Helping the system stay well within the sub-10ms requirement for high-end AR/VR.
  • High Accuracy: Despite the massive data loss, gaze error remains below the 1.0° industry standard.

Conclusion and Future Work

BlissCam demonstrates that in-sensor processing can significantly alleviate the data movement bottleneck in eye tracking. While highly effective, future work will address temporal dependencies in ROI prediction and leakage power in in-sensor buffers. The principles of learned in-sensor sparse sampling could potentially extend to other AR/VR tasks where data reduction is paramount.

Summary updated

BibTeX
@inproceedings{feng2024blisscam,
  author = {Yu Feng and Tianrui Ma and Yuhao Zhu and Xuan Zhang},
  title = {{BlissCam: Boosting Eye Tracking Efficiency with Learned In-Sensor Sparse Sampling}},
  booktitle = {ISCA},
  year = {2024},
  doi = {10.1109/isca59077.2024.00094},
}
Figure