EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream

Viktor Rudnev, Vladislav Golyanik, Jiayi Wang, Hans-Peter Seidel, Franziska Mueller, Mohamed Elgharib, Christian Theobalt

Abstract

3D hand pose estimation from monocular video is a critical challenge for immersive technologies like AR/VR. We present EventHands, the first deep learning approach to reconstruct 3D hand poses using a single event camera—an asynchronous vision sensor that reacts to brightness changes rather than capturing full frames. EventHands achieves an unprecedented temporal resolution of 1000 Hz with low data throughput, enabling real-time tracking of rapid motions that typically cause traditional RGB cameras to fail due to motion blur.

The Challenge of High-Speed Tracking

Standard RGB and depth cameras are limited by their frame-based nature, typically operating at 30–60 fps. When a hand moves quickly, these sensors produce significant motion blur, leading to tracking failures. Event cameras (like the DAVIS240C) solve this by reporting pixel-level changes with microsecond latency. However, event data is sparse and irregular, meaning standard computer vision architectures cannot be applied out of the box. EventHands bridges this gap by introducing a novel representation and a robust neural pipeline.

Methodology: The EventHands Pipeline

To process asynchronous event streams with deep neural networks, we developed a specialized pipeline centered on a new representational innovation.

The EventHands pipeline: from raw event stream to filtered 3D hand pose.

Locally-Normalised Event Surfaces (LNES)

The core of our method is the LNES representation. Instead of simply counting events, LNES encodes events into a 2D image where pixel values represent window-normalized timestamps. This preserves the relative timing and direction of motion, providing the network with rich temporal information within a single input.

Neural Regression and Filtering

We utilize a ResNet-18 backbone to regress the 3D hand pose parameters, specifically predicting the global rotation, translation, and shape parameters of the MANO parametric hand model. To combat jitter and ensure smooth transitions, a constant-velocity Kalman filter is applied to the network's raw outputs. Because annotating real-world 3D event data is difficult, the model is trained on a massive synthetic dataset (100 hours of simulated data) generated by a custom GPU-based simulator.

Key Results and Performance

EventHands demonstrates superior performance in speed, accuracy, and robustness compared to traditional frame-based methods.

Comparison between input event representations (LNES) and the predicted 3D hand poses.
  • Unprecedented Speed: The system achieves an inference speed of 1000 Hz (1ms per pose), an order of magnitude faster than state-of-the-art RGB methods.
  • High Accuracy: On synthetic tests, EventHands achieved a 3D-AUC of 0.85. In real-world evaluations, it significantly outperformed existing RGB methods during high-speed movements where motion blur rendered traditional sensors ineffective.
  • Low-Light Robustness: Qualitative results show the system successfully tracking hands in near-total darkness, where RGB sensors only capture black frames.

Limitations and Future Work

While EventHands sets a new benchmark for speed, it currently assumes a static background. Moving the camera creates background noise that can interfere with the hand signal. Future work will explore handling hand-object interactions and replacing the Kalman filter with learned recurrent models (like LSTMs) to better capture complex human dynamics.

Summary updated

Abstract

3D hand pose estimation from monocular videos is a long-standing and challenging problem, which is now seeing a strong upturn. In this work, we address it for the first time using a single event camera, i.e., an asynchronous vision sensor reacting on brightness changes. Our EventHands approach has characteristics previously not demonstrated with a single RGB or depth camera such as high temporal resolution at low data throughputs and real-time performance at 1000 Hz. Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams. We thus design a new neural approach which accepts a new event stream representation suitable for learning, which is trained on newly-generated synthetic event streams and can generalise to real data. Experiments show that EventHands outperforms recent monocular methods using a colour (or depth) camera in terms of accuracy and its ability to capture hand motions of unprecedented speed. Our method, the event stream simulator and the dataset are publicly available (see https://4dqv.mpi-inf.mpg.de/EventHands/).
BibTeX
@inproceedings{rudnev2021eventhands,
  author = {Viktor Rudnev and Vladislav Golyanik and Jiayi Wang and Hans-Peter Seidel and Franziska Mueller and Mohamed Elgharib and Christian Theobalt},
  title = {{EventHands: Real-Time Neural 3D Hand Pose Estimation from an Event Stream}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=fPOSLznZUob},
}
Figure