Differentiable Event Stream Simulator for Non-Rigid 3D Tracking

Jalees Nehvi, Vladislav Golyanik, Franziska Mueller, Hans-Peter Seidel, Mohamed Elgharib, Christian Theobalt

Abstract

This paper introduces the first differentiable simulator of event streams—asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects, such as human hands, isometric surfaces, and general watertight meshes, from event streams by leveraging an analysis-by-synthesis principle. Unlike previous event-based tracking methods that rely on explicit event trajectories or large-scale datasets, our approach requires no such pre-processing or data and can be readily applied to incoming event streams. Our experiments demonstrate that the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors, particularly in high-speed scenarios.

Introduction

Standard RGB cameras capture frames at fixed intervals, often resulting in motion blur when objects move quickly. Event cameras, or neuromorphic sensors, offer a solution by reporting only per-pixel brightness changes with microsecond precision and high dynamic range. However, the sparse, asynchronous nature of event data makes traditional 3D tracking difficult. Tracking non-rigid objects—those that bend, stretch, or fold—is especially challenging due to the loss of depth information and frequent occlusions. This research presents a method to track general deforming objects directly from raw event streams in an unsupervised manner.

Methodology: Analysis-by-Synthesis

The core of the proposed approach is an Analysis-by-Synthesis framework. Rather than inferring 3D shape directly from events, the system utilizes a 3D template and simulates the events that would be generated by its motion.

Overview of the differentiable event stream simulator and the analysis-by-synthesis optimization framework.

The Differentiable Event Simulator

The simulator pipeline consists of several key stages:

  1. Differentiable Rendering: Two grayscale images of a 3D mesh are rendered at consecutive timestamps.
  2. Difference Calculation: The per-pixel brightness difference between these images is computed.
  3. Smooth Thresholding: To maintain differentiability, the standard hard threshold of event cameras is replaced with a sigmoid-like smooth approximation ($g(x)$). This allows gradients to flow through the thresholding operation.
  4. Optimization: The 3D model's parameters (pose, rotation, and vertex positions) are updated via gradient descent to minimize the difference between simulated and observed events.

The objective function incorporates a "Data Term" that matches events and penalizes "no-events" where none should occur, alongside "Geometric Regularizers" to maintain the realistic structure of the mesh.

Experimental Evaluation

The method was evaluated using both real-world and synthetic datasets:

  • Real-world Data: Captured with a DAVIS 240C event camera, featuring human hands, deforming paper, and a bouncing volleyball.
  • Synthetic Data: Generated in Blender to provide ground truth for quantitative analysis, including sequences of a hand (306 frames), paper (250 frames), and a ball (200 frames).

Results and Key Insights

The proposed method demonstrates significant improvements over traditional RGB-based tracking. In synthetic hand sequences, it achieved a 3D joint error of 0.074, compared to 0.191 for the state-of-the-art HandGraphCNN. This is largely because event cameras remain crisp during high-speed motion where RGB cameras suffer from blur.

Qualitative results for non-rigid tracking of a real hand and a bouncing ball.

A critical finding was the importance of the "No-Event" term in the objective function. Penalizing the model for generating events in regions where the real camera observed silence provides the necessary constraints to prevent the 3D shape from expanding unrealistically. Furthermore, the method is highly general; it can be applied to any object given a basic 3D template, without requiring a massive labeled training set.

Conclusion and Future Work

While the current implementation provides high-fidelity reconstructions, it is not yet real-time, taking approximately 1.5 minutes per frame on an NVIDIA V100 GPU. It also requires an accurate initial 3D state for the template. Future research will focus on improving computational efficiency and potentially integrating this differentiable simulator into deep learning models to enhance their understanding of event-based data for real-time AR/VR applications.

Summary updated

Abstract

This paper introduces the first differentiable simulator of event streams, i.e., streams of asynchronous brightness change signals recorded by event cameras. Our differentiable simulator enables non-rigid 3D tracking of deformable objects (such as human hands, isometric surfaces and general watertight meshes) from event streams by leveraging an analysis-by-synthesis principle. So far, event-based tracking and reconstruction of non-rigid objects in 3D, like hands and body, has been either tackled using explicit event trajectories or large-scale datasets. In contrast, our method does not require any such processing or data, and can be readily applied to incoming event streams. We show the effectiveness of our approach for various types of non-rigid objects and compare to existing methods for non-rigid 3D tracking. In our experiments, the proposed energy-based formulations outperform competing RGB-based methods in terms of 3D errors. The source code and the new data are publicly available.
BibTeX
@inproceedings{nehvi2021differentiable,
  author = {Jalees Nehvi and Vladislav Golyanik and Franziska Mueller and Hans-Peter Seidel and Mohamed Elgharib and Christian Theobalt},
  title = {{Differentiable Event Stream Simulator for Non-Rigid 3D Tracking}},
  booktitle = {Workshops @ CVPR},
  year = {2021},
  url = {https://openreview.net/forum?id=d7zlp4azrEP},
}
Figure