End-to-end Learning of Object Motion Estimation from Retinal Events for Event-based Object Tracking

Haosheng Chen, David Suter, Qiangqiang Wu, Hanzi Wang

Abstract

Event cameras, which are asynchronous bio-inspired vision sensors, have shown great potential in computer vision and artificial intelligence. However, the application of event cameras to object-level motion estimation or tracking is still in its infancy. The main idea behind this work is to propose a novel deep neural network to learn and regress a parametric object-level motion/transform model for event-based object tracking. To achieve this goal, we propose a synchronous Time-Surface with Linear Time Decay (TSLTD) representation, which effectively encodes the spatio-temporal information of asynchronous retinal events into TSLTD frames with clear motion patterns. We feed the sequence of TSLTD frames to a novel Retinal Motion Regression Network (RMRNet) to perform an end-to-end 5-DoF object motion regression. Our method is compared with state-of-the-art object tracking methods, that are based on conventional cameras or event cameras. The experimental results show the superiority of our method in handling various challenging environments such as fast motion and low illumination conditions.

Introduction

Standard video cameras capture frames at fixed intervals, which often results in motion blur during high-speed movement and a loss of detail in high dynamic range (HDR) scenes. Event cameras, such as the DAVIS sensor, offer a bio-inspired alternative by reporting pixel-level brightness changes (events) asynchronously with microsecond precision. While these sensors excel in extreme conditions, their sparse and irregular data output is difficult for standard deep learning architectures to process. This work addresses this gap by introducing RMRNet, a framework that regresses 5-DoF object motion directly from a novel event representation.

Methodology

The proposed pipeline consists of two primary components: a motion-aware data representation and a specialized regression network.

TSLTD Representation

To bridge the gap between asynchronous events and convolutional neural networks (CNNs), we aggregate events into Time-Surface with Linear Time Decay (TSLTD) frames. Unlike traditional exponential decay surfaces, TSLTD uses linear decay to produce clear intensity gradients. These gradients serve as visual signatures of an object's velocity and direction, effectively encoding motion information into a format suitable for spatial feature extraction.

RMRNet Architecture

The Retinal Motion Regression Network (RMRNet) processes a sequence of five TSLTD patches. The architecture comprises:

  1. CNN Layers: Four convolutional layers extract spatial features from the motion patterns in the TSLTD frames.
  2. LSTM Module: A Long Short-Term Memory (LSTM) module integrates these features over time, capturing the temporal evolution of the object's movement.
  3. 5-DoF Regression: The network outputs five transformation parameters: horizontal displacement ($d_x$), vertical displacement ($d_y$), rotation ($\theta$), and scaling ($s_x, s_y$).
Overview of the RMRNet pipeline, from retinal event integration to 5-DoF bounding box regression.

Experimental Evaluation

We evaluated RMRNet on the Event Camera Dataset (ECD) and the Extreme Event Dataset (EED), comparing it against state-of-the-art (SOTA) trackers including SiamFC, ECO, and ATOM.

Performance in Extreme Conditions

RMRNet demonstrated significant advantages in scenarios where traditional frame-based cameras fail:

  • Fast Motion: On the fast_drone sequence, RMRNet achieved an Average Overlap Rate (AOR) of 0.892, whereas traditional trackers struggled with motion blur.
  • Low Illumination: The sensor's high dynamic range allowed RMRNet to maintain high tracking accuracy in dark environments.
  • Occlusions: In the EED occlusions sequence, RMRNet reached an AOR of 0.716, nearly doubling the performance of the next best competitor, ATOM (0.387).
Qualitative comparison of RMRNet against state-of-the-art trackers across various challenging sequences.

Efficiency

Despite the inclusion of an LSTM, RMRNet remains computationally efficient. It processes an object pair in approximately 38.57 ms on a standard GTX 1080 GPU, supporting real-time tracking applications.

Conclusion and Future Work

RMRNet provides a robust, end-to-end solution for event-based object tracking, particularly in high-speed and low-light environments. A current limitation is the handling of dense dynamic clutter, as seen in the what_is_background sequence, where foreground objects disrupt motion patterns. Future research will focus on improving robustness to dynamic occlusions and extending the model to more complex 3D motion estimation.

Summary updated

Abstract

Event cameras, which are asynchronous bio-inspired vision sensors, have shown great potential in computer vision and artificial intelligence. However, the application of event cameras to object-level motion estimation or tracking is still in its infancy. The main idea behind this work is to propose a novel deep neural network to learn and regress a parametric object-level motion/transform model for event-based object tracking. To achieve this goal, we propose a synchronous Time-Surface with Linear Time Decay (TSLTD) representation, which effectively encodes the spatio-temporal information of asynchronous retinal events into TSLTD frames with clear motion patterns. We feed the sequence of TSLTD frames to a novel Retinal Motion Regression Network (RMRNet) to perform an end-to-end 5-DoF object motion regression. Our method is compared with state-of-the-art object tracking methods, that are based on conventional cameras or event cameras. The experimental results show the superiority of our method in handling various challenging environments such as fast motion and low illumination conditions.
BibTeX
@inproceedings{chen2020end,
  author = {Haosheng Chen and David Suter and Qiangqiang Wu and Hanzi Wang},
  title = {{End-to-end Learning of Object Motion Estimation from Retinal Events for Event-based Object Tracking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6625},
}
Figure