Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

Jalees Nehvi, Vladislav Golyanik, Franziska Mueller, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

Abstract

This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available.

Introduction

Event cameras represent a radical departure from standard frame-based cameras. Instead of capturing full image frames at fixed intervals, they record asynchronous "events"—pixel-level changes in brightness—with microsecond-level precision. This makes them immune to motion blur and excellent for high-speed tracking. However, 3D tracking with event cameras is difficult because they do not provide steady intensity values, causing traditional computer vision algorithms to fail. Previous attempts at non-rigid 3D tracking with events usually relied on massive supervised datasets or were limited to specific classes. This paper addresses the lack of a general-purpose, unsupervised framework for tracking any deforming 3D object using only an event stream.

Methodology

The core of the research is a differentiable event stream simulator used within an analysis-by-synthesis loop. This framework allows for the estimation of 3D parameters by comparing simulated events to real-world observations.

Overview of the proposed analysis-by-synthesis framework for non-rigid 3D tracking from event streams.

As shown in the framework overview, the system starts with a 3D template (a mesh or a parametric model). The simulator renders two "virtual" images of the object at consecutive timestamps. By subtracting these images, it calculates brightness changes. To convert these changes into differentiable "events," the authors use a novel, smooth thresholding function. This allows the system to calculate exactly how to tweak the 3D model's parameters to match the real-world event stream. The optimization minimizes an energy function that includes a data term, a vital "no-event" term (penalizing the simulator for creating events where the camera saw none), and geometric regularizers to ensure mesh smoothness.

Experiments and Results

The authors evaluated their method on both real-world and synthetic data, including human hands, deforming paper, and a volleyball.

Qualitative comparison of 3D reconstruction for a deforming checkerboard surface against state-of-the-art methods.

Key results from the evaluation include:

  • Hands: On synthetic hand sequences, the method achieved a 3D joint error of 0.074, nearly 2.5x lower than the state-of-the-art RGB-based HandGraphCNN (0.191).
  • Deformable Surfaces: For deforming paper sequences, it outperformed existing methods like DDD and IsMo-GAN, achieving the lowest 3D reconstruction error (0.232).
  • Fast Motion Robustness: In scenarios where objects move quickly, standard RGB cameras produce motion blur that ruins tracking. Because event cameras have a temporal resolution of 1µs, this method maintains accuracy where RGB methods fail.
  • Efficiency: Event data is highly efficient; for a 36-second recording, the event data took up only 23 MB, compared to 110 MB for the corresponding RGB video.

Discussion and Future Work

While the method demonstrates high accuracy and generality, it currently faces limitations in processing speed, taking about 1.5 minutes per frame on an NVIDIA V100 GPU. Additionally, the system assumes the initial 3D pose is known. Future work suggests integrating this differentiable simulator into deep learning pipelines as a "known-physics" layer, potentially combining the speed of neural networks with the precision of physics-based simulation.

Summary updated

Abstract

This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available.
BibTeX
@inproceedings{nehvi2021differentiable,
  author = {Jalees Nehvi and Vladislav Golyanik and Franziska Mueller and Hans-Peter Seidel and Mohamed Elgharib and Christian Theobalt},
  title = {{Differentiable Event Stream Simulator for Non-Rigid 3D Tracking}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=d7zlp4azrEP},
}
Figure