Abstract
3D hand pose estimation from monocular video is a critical challenge for immersive technologies like AR/VR. We present EventHands, the first deep learning approach to reconstruct 3D hand poses using a single event camera—an asynchronous vision sensor that reacts to brightness changes rather than capturing full frames. EventHands achieves an unprecedented temporal resolution of 1000 Hz with low data throughput, enabling real-time tracking of rapid motions that typically cause traditional RGB cameras to fail due to motion blur.
The Challenge of High-Speed Tracking
Standard RGB and depth cameras are limited by their frame-based nature, typically operating at 30–60 fps. When a hand moves quickly, these sensors produce significant motion blur, leading to tracking failures. Event cameras (like the DAVIS240C) solve this by reporting pixel-level changes with microsecond latency. However, event data is sparse and irregular, meaning standard computer vision architectures cannot be applied out of the box. EventHands bridges this gap by introducing a novel representation and a robust neural pipeline.
Methodology: The EventHands Pipeline
To process asynchronous event streams with deep neural networks, we developed a specialized pipeline centered on a new representational innovation.
Locally-Normalised Event Surfaces (LNES)
The core of our method is the LNES representation. Instead of simply counting events, LNES encodes events into a 2D image where pixel values represent window-normalized timestamps. This preserves the relative timing and direction of motion, providing the network with rich temporal information within a single input.
Neural Regression and Filtering
We utilize a ResNet-18 backbone to regress the 3D hand pose parameters, specifically predicting the global rotation, translation, and shape parameters of the MANO parametric hand model. To combat jitter and ensure smooth transitions, a constant-velocity Kalman filter is applied to the network's raw outputs. Because annotating real-world 3D event data is difficult, the model is trained on a massive synthetic dataset (100 hours of simulated data) generated by a custom GPU-based simulator.
Key Results and Performance
EventHands demonstrates superior performance in speed, accuracy, and robustness compared to traditional frame-based methods.
- Unprecedented Speed: The system achieves an inference speed of 1000 Hz (1ms per pose), an order of magnitude faster than state-of-the-art RGB methods.
- High Accuracy: On synthetic tests, EventHands achieved a 3D-AUC of 0.85. In real-world evaluations, it significantly outperformed existing RGB methods during high-speed movements where motion blur rendered traditional sensors ineffective.
- Low-Light Robustness: Qualitative results show the system successfully tracking hands in near-total darkness, where RGB sensors only capture black frames.
Limitations and Future Work
While EventHands sets a new benchmark for speed, it currently assumes a static background. Moving the camera creates background noise that can interfere with the hand signal. Future work will explore handling hand-object interactions and replacing the Kalman filter with learned recurrent models (like LSTMs) to better capture complex human dynamics.
Summary updated