EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams

Christen Millerdurai, Hiroyasu Akada, Jian Wang, Diogo Luvizon, Christian Theobalt, Vladislav Golyanik

TL;DR

EventEgo3D (EE3D) is the first framework to achieve real-time 3D human motion capture using a single, head-mounted fisheye event camera. By leveraging the high temporal resolution and dynamic range of event streams, the system maintains high accuracy (140Hz update rate) even during rapid movements and in extreme low-light conditions where traditional RGB cameras typically fail.

What Problem Does This Paper Address?

Egocentric 3D human pose estimation—tracking a person's body movements from a camera mounted on their own head—is a cornerstone for AR/VR, gaming, and personal assistance. However, current solutions rely almost exclusively on standard RGB cameras. These sensors have three major Achilles' heels in mobile scenarios:

  1. Motion Blur: Fast limb or head movements result in blurry frames that break tracking.
  2. Dynamic Range: Moving between bright outdoors and dim indoors often leads to over- or under-exposure.
  3. Power Consumption: Constantly processing high-resolution RGB frames is computationally expensive and drains mobile batteries.

Event cameras, which only record per-pixel brightness changes asynchronously, offer a solution. They provide microsecond-level temporal resolution and massive dynamic range. However, they introduce a new challenge: because the camera is head-mounted, every head movement generates a "storm" of background events, making it difficult to distinguish the wearer's limbs from the environment.

Methods

The authors propose EventEgo3D (EE3D), an end-to-end neural architecture designed to process event streams for pose estimation.

Overview of the EE3D architecture, showing the EPM and REPM modules.

The pipeline consists of two primary components:

  • Egocentric Pose Module (EPM): This module takes Locally Normalised Event Surfaces (LNES)—a compact 2D representation of event data—and uses a U-Net-based architecture to estimate 2D joint heatmaps. A subsequent "lifting" block regresses these heatmaps into 3D joint coordinates.
  • Residual Event Propagation Module (REPM): This is the core innovation for handling egocentric noise. It uses a segmentation decoder to identify the wearer's body and a confidence decoder to weight events. By maintaining a frame buffer, it propagates "event history," allowing the system to track the user even when they are momentarily still (and thus generating few new events) or when background noise is high.

The hardware setup involves a custom-built HMD featuring a fisheye event camera with a 190° field of view, connected to a laptop housed in a backpack for real-time processing.

Datasets

Because no egocentric event-based pose datasets existed, the authors created two new benchmarks:

  • EE3D-S (Synthetic): A large-scale dataset generated using the VID2E simulator. It includes 946 motion sequences and over 6 million 3D poses, featuring diverse lighting and background augmentations.
  • EE3D-R (Real): Recorded with their custom HMD in a multi-view studio. It features 12 subjects performing various actions (boxing, dancing, crawling) with ground-truth 3D poses provided by a professional motion capture system.

Key Insights and Ideas

  • Events as a Sparsity Advantage: Unlike RGB cameras that process every pixel regardless of change, event cameras naturally focus on motion. EE3D exploits this to achieve a high update rate of 140Hz on a mobile GPU.
  • Taming the Background: The REPM module effectively solves the "moving camera" problem. By learning to segment the user and propagate temporal context, the network learns to ignore the "visual chatter" caused by head rotations.
  • LNES Representation: Using Locally Normalised Event Surfaces allows the use of efficient CNN layers (like Blaze blocks) while remaining independent of the raw event count, ensuring stable performance regardless of how fast the user moves.

Key Results

EE3D significantly outperforms existing RGB-based methods that were adapted to handle event data.

Qualitative comparison showing EE3D's superior tracking compared to adapted state-of-the-art methods.
  • Accuracy: On the real-world EE3D-R dataset, the method achieved a Mean Per Joint Position Error (MPJPE) of 107.30 mm, a 6% improvement over the nearest event-based competitor and a massive improvement over adapted RGB methods (which erred by over 130-170 mm).
  • Speed: The system supports a 140Hz pose update rate, making it suitable for low-latency applications like VR.
  • Efficiency: The model is remarkably lightweight, using only 1.25M parameters—nearly 60x fewer than some RGB-based counterparts.

Limitations and Future Work

While EE3D handles background noise well, the authors note a remaining domain gap between synthetic and real data; models trained only on synthetic data perform poorly until fine-tuned on real event streams.

Future work could explore:

  • Moving the processing from a backpack-mounted laptop to on-chip HMD hardware.
  • Extending the framework to multi-person tracking from an egocentric perspective.
  • Integrating the system with downstream tasks like egocentric object interaction or navigation.

Summary updated

Abstract

Monocular egocentric 3D human motion capture is a challenging and actively researched problem. Existing methods use synchronously operating visual sensors (e.g. RGB cameras) and often fail under low lighting and fast motions, which can be restricting in many applications involving head-mounted devices. In response to the existing limitations, this paper 1) introduces a new problem, i.e., 3D human motion capture from an egocentric monocular event camera with a fisheye lens, and 2) proposes the first approach to it called EventEgo3D (EE3D). Event streams have high temporal resolution and provide reliable cues for 3D human motion capture under high-speed human motions and rapidly changing illumination. The proposed EE3D framework is specifically tailored for learning with event streams in the LNES representation, enabling high 3D reconstruction accuracy. We also design a prototype of a mobile head-mounted device with an event camera and record a real dataset with event observations and the ground-truth 3D human poses (in addition to the synthetic dataset). Our EE3D demonstrates robustness and superior 3D accuracy compared to existing solutions across various challenging experiments while supporting real-time 3D pose update rates of 140Hz.
BibTeX
@misc{millerdurai2024eventego,
  author = {Christen Millerdurai and Hiroyasu Akada and Jian Wang and Diogo Luvizon and Christian Theobalt and Vladislav Golyanik},
  title = {{EventEgo3D: 3D Human Motion Capture from Egocentric Event Streams}},
  year = {2024},
  eprint = {2404.08640},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2404.08640},
}

Loading related content...

Figure