Abstract
Eye tracking is a critical component for modern Augmented and Virtual Reality (AR/VR) systems, enabling features such as foveated rendering and gaze-based interaction. However, existing systems are constrained by significant power consumption and latency, often exceeding 2W and 15ms respectively. BlissCam addresses these bottlenecks by introducing "in-sensor sparse sampling." By integrating motion detection and random sampling directly into the image sensor hardware, BlissCam discards 95% of pixel data at the source. This approach achieves an 8.2× reduction in energy consumption and a 1.4× improvement in end-to-end latency while maintaining high tracking accuracy (gaze error < 1.0°).
Introduction: The Power and Latency Wall
The "power and latency wall" in current AR/VR eye trackers stems from a frontend bottleneck: the image sensor. Standard sensors read out every pixel, forcing the system to move and process massive amounts of redundant data. While many researchers focus on optimizing backend algorithms, BlissCam co-designs the sensor hardware and tracking algorithms to "thin out" data during capture.
Methods: In-Sensor and Off-Sensor Co-Design
The BlissCam pipeline is divided into in-sensor and off-sensor components to maximize efficiency without sacrificing robustness.
1. In-Sensor Sparse Sampling
Using a modified Digital Pixel Sensor (DPS) architecture, BlissCam performs three key steps directly on the chip:
- Eventification: The sensor compares the current frame to the previous one. Since the camera is fixed relative to the head, the background is stationary, allowing the sensor to focus only on moving parts (the eye).
- ROI Prediction: A tiny, in-sensor Neural Processing Unit (NPU) uses these "events" to predict a Region-of-Interest (ROI) box around the eye.
- Random Sampling: Within that ROI, the sensor only quantizes and reads out a small fraction (~5%) of pixels chosen at random.
Hardware efficiency is achieved by time-multiplexing existing Single-Slope ADC components and per-pixel SRAM, resulting in only a 5.8% increase in silicon area.
2. Off-Sensor Robust Segmentation
The host processor receives sparse pixel data resembling a "starry night." BlissCam utilizes a Vision Transformer (ViT) for segmentation. Unlike traditional CNNs, which rely on local spatial continuity, the ViT’s attention mechanism effectively relates distant pixels, making it naturally robust to high levels of data sparsity.
Key Results
Evaluation on the OpenEDS dataset demonstrates significant improvements over traditional pipelines:
- 95% Pixel Volume Reduction: Achieving a 20× compression rate at the source.
- 8.2× Energy Reduction: Primarily due to reduced activity in the power-hungry readout chain and MIPI interfaces.
- 1.4× Latency Reduction: Helping the system stay well within the sub-10ms requirement for high-end AR/VR.
- High Accuracy: Despite the massive data loss, gaze error remains below the 1.0° industry standard.
Conclusion and Future Work
BlissCam demonstrates that in-sensor processing can significantly alleviate the data movement bottleneck in eye tracking. While highly effective, future work will address temporal dependencies in ROI prediction and leakage power in in-sensor buffers. The principles of learned in-sensor sparse sampling could potentially extend to other AR/VR tasks where data reduction is paramount.
Summary updated