Robust Feature Tracking in DVS Event Stream using Bézier Mapping

Hochang Seok, Jongwoo LimWACV 2020 · OpenReview

TL;DR

This paper introduces a novel method for robustly tracking features in event camera data by aligning events using Bézier curves to combat motion blur. This improved alignment, coupled with a temporal template update mechanism, leads to significantly more accurate and persistent feature tracking, ultimately enhancing applications like 3D pose estimation.

What Problem Does This Paper Address?

Event cameras, unlike traditional frame-based cameras, capture asynchronous intensity changes at each pixel, providing a continuous stream of events with precise timestamps. This offers advantages like low latency and high dynamic range, making them ideal for high-speed motion. However, this asynchronous nature means there are no "frames" in the conventional sense. Many computer vision tasks, such as 3D pose estimation, rely on tracking 2D feature point trajectories. Existing methods for event cameras often stack events within fixed time windows to create "event images." The primary challenge is that camera motion during these time windows causes misalignment and blur in these event images, degrading the accuracy of feature detection and tracking. This paper aims to overcome this by developing a more robust event alignment and feature tracking technique.

Methods

The core innovation lies in two main components:

  • Bézier Curve Event Alignment: Instead of simply stacking events or aligning them along straight lines, the authors propose using quadratic Bézier curves to align events within a time window. The Bézier curve's control points are estimated by maximizing the variance of the resulting aligned event patch. Higher variance indicates better contrast and less blur, thus indicating a more accurate alignment that accounts for camera motion. This is a departure from previous methods that either ignored motion blur or used simpler linear alignments.
  • Long-Term Feature Tracking with Temporal Template Update: To ensure robust tracking over extended periods, the paper introduces a temporal template update mechanism. Instead of using a single initial patch, the algorithm maintains a "template patch" that is continuously updated using events from the current stream with an exponential decay. This integrated template captures richer texture information and is more resilient to noise and missing features that can occur in individual aligned event patches, especially during challenging motion.

Datasets

The paper evaluates its method on:

  • Synthetic Datasets: Six sequences (planes, desk, gate, wall, roof, drum) generated using the ESIM simulator, providing ground truth feature trajectories.
  • Real Datasets: Three sequences from a public dataset (shapes, poster, boxes) with ground truth information.

Key Insights and Ideas

  • Bézier curves are superior to straight lines for event alignment: The paper demonstrates that Bézier curves can model more complex motion trajectories than simple straight lines, leading to significantly sharper event images and higher variance.
  • Maximizing variance is a good proxy for alignment quality: The authors leverage the principle that well-aligned event patches, reflecting scene structure, will exhibit higher variance (contrast) compared to blurred or misaligned ones.
  • Temporal template integration is crucial for long-term robustness: Relying solely on short-term aligned event patches is insufficient. Continuously updating a template patch with historical event data allows the tracker to maintain a richer representation of the feature, preventing drift and improving persistence.
  • Improved alignment directly benefits downstream applications: The paper shows that the enhanced feature tracking accuracy directly translates to more reliable 3D pose estimation.

Key Results

  • Event Alignment: The proposed Bézier curve alignment achieved significantly higher mean variance (23.6% and 279.02% higher than straight-line and stacking methods, respectively) on test datasets, indicating better alignment quality.
  • Feature Tracking (Synthetic Data): The proposed method tracked features significantly longer (e.g., feature age was 6 times higher for a 3-pixel threshold on the planes dataset) and with lower RMSE compared to the state-of-the-art Zhu et al. [29]. The method without temporal update (TU) also showed improvement over Zhu et al., highlighting the benefit of Bézier alignment alone.
  • Feature Tracking (Real Data): Similar improvements in tracking length and RMSE were observed on real datasets, with the proposed method outperforming Zhu et al. on shapes and boxes, and showing comparable or better results on the poster sequence.
  • 3D Pose Estimation: The method demonstrated significantly more accurate and stable 3D pose estimation compared to Zhu et al., with higher inlier ratios and trajectories closely matching ground truth, even in challenging 6-DOF motion scenarios.

Limitations and Future Work

  • Feature Detection Initialization: The paper uses a standard Harris corner detector for initial feature point detection. The authors mention plans to investigate better ways for feature detection specifically tailored for event streams.
  • Poster Sequence Performance: While generally strong, the performance on the "poster" sequence was less stable, attributed to lower event density and features moving out of the image.
  • Computational Cost: While not explicitly detailed as a limitation, the Bézier curve estimation and temporal template update likely add computational overhead compared to simpler methods. Future work could focus on optimizing these aspects.
  • Public Availability: The authors mention that rendering codes for synthetic datasets will be made public. The implementation of the proposed algorithm is not explicitly stated as public.

Summary updated

Abstract

Unlike conventional cameras, event cameras capture the intensity changes at each pixel with very little delay. Such changes are recorded as an event stream with their positions, timestamps, and polarities continuously, thus there is no notion of `frame' as in conventional cameras. As many applications including 3D pose estimation use 2D trajectories of feature points, it is necessary to detect and track the feature points robustly and accurately in a continuous event stream. In conventional feature tracking algorithms for event streams, the events in fixed time intervals are converted into the event images by stacking the events at their pixel locations, and the features are tracked in the event images. Such simple stacking of events yields blurry event images due to the camera motion, and it can significantly degrade the tracking quality. We propose to align the events in the time intervals along Bézier curves to minimize the misalignment. Since the camera motion is unknown, the Bézier curve is estimated to maximize the variance of the warped event pixels. Instead of the initial patches for tracking, we use the temporally integrated template patches, as it captures rich texture information from accurately aligned events. Extensive experimental evaluations in 2D feature tracking as well as 3D pose estimation show that our method significantly outperforms the conventional approaches.
BibTeX
@inproceedings{seok2020robust,
  author = {Hochang Seok and Jongwoo Lim},
  title = {{Robust Feature Tracking in DVS Event Stream using Bézier Mapping}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
  url = {https://openreview.net/forum?id=9OypVVsoqV},
}
Figure