Abstract
3D hand pose estimation from monocular videos is a long-standing and challenging problem, which is now seeing a strong upturn. In this work, we address it for the first time using a single event camera, i.e., an asynchronous vision sensor reacting on brightness changes. Our EventHands approach has characteristics previously not demonstrated with a single RGB or depth camera such as high temporal resolution at low data throughputs and real-time performance at 1000 Hz. Due to the different data modality of event cameras compared to classical cameras, existing methods cannot be directly applied to and re-trained for event streams. We thus design a new neural approach which accepts a new event stream representation suitable for learning, which is trained on newly-generated synthetic event streams and can generalise to real data. Experiments show that EventHands outperforms recent monocular methods using a colour (or depth) camera in terms of accuracy and its ability to capture hand motions of unprecedented speed. Our method, the event stream simulator and the dataset are publicly available at https://4dqv.mpi-inf.mpg.de/EventHands/.
Introduction
Traditional 3D hand tracking relies on standard RGB or depth cameras, which typically operate at 30–60 frames per second (fps). This frame-based paradigm suffers from two major limitations: motion blur and latency. Rapid hand movements result in blurry images that cause tracking algorithms to fail, while the high data throughput required for high-resolution video makes ultra-fast processing computationally expensive.
Event cameras (neuromorphic sensors) offer a solution by asynchronously reporting pixel-level brightness changes ("events") with microsecond precision. However, because event data is sparse and lacks traditional texture, existing 3D hand pose estimation models cannot be used directly. EventHands bridges this gap by introducing a specialized representation and training pipeline for event streams, enabling real-time 3D reconstruction at 1000 Hz.
Methodology
To enable neural learning from asynchronous events, the researchers introduced Locally-Normalised Event Surfaces (LNES). Unlike standard event count images, LNES encodes the relative timestamp of the most recent event at each pixel, preserving the temporal ordering of motion within a time window.
The architecture utilizes a ResNet-18 backbone to regress the parameters of the MANO hand model, outputting global rotation, translation, and pose coefficients. To handle sensor noise and the domain gap between synthetic and real data, a constant-velocity Kalman filter is applied to the predictions, ensuring stable 3D reconstruction over time. Training was conducted on a massive synthetic dataset containing 100 hours of simulated event data, overcoming the difficulty of obtaining ground-truth 3D labels for real-world event streams.
Results and Key Insights
EventHands achieves a processing rate of 1000 Hz on a single GPU, an order of magnitude faster than most state-of-the-art image-based methods. On real-world data captured with a DAVIS240C event camera, the system achieved a 0.77 2D-AUCp (Area Under the Curve for palm-normalized 2D PCK).
Key findings include:
- Temporal Importance: LNES significantly outperforms simpler representations for long time windows, proving that the timing of events is more critical for 3D reconstruction than event density.
- Robustness: The system successfully tracks extremely fast motions and performs in low-light environments where traditional RGB cameras fail due to underexposure or motion blur.
- Generalization: Despite being trained entirely on synthetic data, the model generalizes effectively to real-world streams, suggesting that event-based abstractions help bridge the sim-to-real gap.
Limitations and Future Work
While a significant advancement, EventHands currently assumes a mostly static background; significant camera movement can introduce noise that degrades tracking. Furthermore, the model is optimized for single-hand scenarios. Future research will explore complex hand-object interactions, multi-hand tracking, and hybrid systems that combine event data with sparse RGB anchor frames for enhanced detail.
Summary updated