Introduction
Traditional frame-based cameras are often inadequate for capturing high-speed motion due to their fixed capture intervals, which result in "blind time" and motion blur. This limitation is particularly problematic for 3D feature tracking in dynamic environments, a critical component for applications in autonomous driving, virtual reality (VR), and augmented reality (AR).
Event cameras offer a promising alternative. These bio-inspired sensors asynchronously report pixel-level brightness changes with microsecond-level temporal resolution, effectively eliminating motion blur. However, most existing event-based research has focused on 2D tracking. This paper introduces E-3DTrack, the first dedicated framework and dataset for high-speed 3D feature tracking using stereo event cameras.
The E-3DTrack Framework
The researchers developed a joint framework that simultaneously predicts 2D motion offsets and 3D spatial positions. The architecture is built upon three primary components designed to handle the unique challenges of asynchronous event data.
1. Motion Compensation Module
As objects move through 3D space, their visual appearance changes due to variations in size, rotation, and shear. To prevent tracker drift, this module applies an affine transform to "correct" the initial feature template at each timestamp, ensuring the tracker remains locked onto the deforming object.
2. Patch Matching via Bi-Polarity Hypergraph
Determining 3D position requires matching patches between stereo cameras to calculate disparity. E-3DTrack employs a "bi-polarity hypergraph" to model high-order correlations. This approach enhances connections between semantically similar and spatially close patches while suppressing distant distractors, significantly improving robustness in complex scenes.
3. Stereo Motion Consistency
A geometric constraint is introduced to link 2D motion in both camera views with changes in disparity. This joint optimization ensures that the predicted 3D trajectories are physically plausible and smooth over time.
E-3DTrack Dataset
To address the lack of high-quality 3D ground-truth data for event cameras, the authors created the E-3DTrack dataset. It consists of 40 high-speed motion scenarios across 1,300 sequences. Ground truth was established at 250 FPS using a hybrid system: an Optitrack motion capture system for trajectory recording and a FARO Quantum ScanArm for high-precision 3D point clouds. Unlike previous datasets, E-3DTrack focuses on dynamic, non-homogeneous 3D motion.
Key Results and Insights
The proposed method demonstrates significant improvements over baseline systems that combine state-of-the-art 2D trackers with standard stereo depth estimators:
- Accuracy: E-3DTrack achieved a 37.5% reduction in Root Mean Squared Error (RMSE) compared to the best baseline.
- Robustness: The system showed a higher "Feature Age" and "Tracked Feature Ratio," indicating more stable long-term tracking.
- Efficiency: With an inference time of approximately 40.3 ms per step, the system is capable of real-time performance at high frequencies.
A key insight from this work is that joint optimization of 2D tracking and depth estimation is superior to sequential processing. By applying geometric constraints, 2D motion can directly inform 3D positioning, leading to more accurate reconstructions of fast-moving scenes.
Limitations and Future Work
While E-3DTrack marks a significant advancement, several areas for future research remain. The current method relies on an initial grayscale template, suggesting a move toward purely event-based initialization as a next step. Additionally, handling extreme, long-duration occlusions and scaling the hypergraph approach to even more cluttered environments with hundreds of moving objects are ongoing challenges.
Summary updated