3D Feature Tracking via Event Camera

Siqi Li, Zhikuan Zhou, Zhou Xue, Yipeng Li, Shaoyi Du, Yue GaoCVPR 2024 · DOI

Introduction

Traditional frame-based cameras are often inadequate for capturing high-speed motion due to their fixed capture intervals, which result in "blind time" and motion blur. This limitation is particularly problematic for 3D feature tracking in dynamic environments, a critical component for applications in autonomous driving, virtual reality (VR), and augmented reality (AR).

Event cameras offer a promising alternative. These bio-inspired sensors asynchronously report pixel-level brightness changes with microsecond-level temporal resolution, effectively eliminating motion blur. However, most existing event-based research has focused on 2D tracking. This paper introduces E-3DTrack, the first dedicated framework and dataset for high-speed 3D feature tracking using stereo event cameras.

Overview of 3D feature tracking using stereo event cameras, showing trajectories of high-speed moving objects.

The E-3DTrack Framework

The researchers developed a joint framework that simultaneously predicts 2D motion offsets and 3D spatial positions. The architecture is built upon three primary components designed to handle the unique challenges of asynchronous event data.

The E-3DTrack architecture, illustrating the motion compensation module, offset estimation, and bi-polarity hypergraph for patch matching.

1. Motion Compensation Module

As objects move through 3D space, their visual appearance changes due to variations in size, rotation, and shear. To prevent tracker drift, this module applies an affine transform to "correct" the initial feature template at each timestamp, ensuring the tracker remains locked onto the deforming object.

2. Patch Matching via Bi-Polarity Hypergraph

Determining 3D position requires matching patches between stereo cameras to calculate disparity. E-3DTrack employs a "bi-polarity hypergraph" to model high-order correlations. This approach enhances connections between semantically similar and spatially close patches while suppressing distant distractors, significantly improving robustness in complex scenes.

3. Stereo Motion Consistency

A geometric constraint is introduced to link 2D motion in both camera views with changes in disparity. This joint optimization ensures that the predicted 3D trajectories are physically plausible and smooth over time.

E-3DTrack Dataset

To address the lack of high-quality 3D ground-truth data for event cameras, the authors created the E-3DTrack dataset. It consists of 40 high-speed motion scenarios across 1,300 sequences. Ground truth was established at 250 FPS using a hybrid system: an Optitrack motion capture system for trajectory recording and a FARO Quantum ScanArm for high-precision 3D point clouds. Unlike previous datasets, E-3DTrack focuses on dynamic, non-homogeneous 3D motion.

Key Results and Insights

The proposed method demonstrates significant improvements over baseline systems that combine state-of-the-art 2D trackers with standard stereo depth estimators:

  • Accuracy: E-3DTrack achieved a 37.5% reduction in Root Mean Squared Error (RMSE) compared to the best baseline.
  • Robustness: The system showed a higher "Feature Age" and "Tracked Feature Ratio," indicating more stable long-term tracking.
  • Efficiency: With an inference time of approximately 40.3 ms per step, the system is capable of real-time performance at high frequencies.

A key insight from this work is that joint optimization of 2D tracking and depth estimation is superior to sequential processing. By applying geometric constraints, 2D motion can directly inform 3D positioning, leading to more accurate reconstructions of fast-moving scenes.

Limitations and Future Work

While E-3DTrack marks a significant advancement, several areas for future research remain. The current method relies on an initial grayscale template, suggesting a move toward purely event-based initialization as a next step. Additionally, handling extreme, long-duration occlusions and scaling the hypergraph approach to even more cluttered environments with hundreds of moving objects are ongoing challenges.

Summary updated

BibTeX
@inproceedings{li2024d,
  author = {Siqi Li and Zhikuan Zhou and Zhou Xue and Yipeng Li and Shaoyi Du and Yue Gao},
  title = {{3D Feature Tracking via Event Camera}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.01795},
}
Figure