End-to-End Learning of Object Motion Estimation from Retinal Events for Event-Based Object Tracking

Haosheng Chen, David Suter, Qiangqiang Wu, Hanzi Wang

Introduction

Traditional computer vision relies on frame-based cameras that capture images at fixed intervals. While effective for static scenes, these sensors often fail in dynamic environments due to motion blur during high-speed movement or dynamic range limitations in high-contrast lighting. Event cameras, such as the DAVIS sensor, offer a bio-inspired alternative by reporting pixel-level brightness changes asynchronously as a stream of "retinal events."

Visualization of raw retinal events, which capture changes in brightness rather than static intensity.

Despite their advantages—high temporal resolution and high dynamic range—event cameras produce sparse, irregular data that is incompatible with standard convolutional neural networks (CNNs). This paper presents an end-to-end deep learning framework designed to estimate 5-degree-of-freedom (5-DoF) object motion directly from these events, enabling robust tracking in extreme conditions.

Methodology: TSLTD and RMRNet

The proposed system bridges the gap between asynchronous events and synchronous deep learning through a two-stage process: encoding and regression.

TSLTD Representation

To transform raw events into a format suitable for CNNs, the authors introduce TSLTD (Time-Surface with Linear Time Decay). Unlike previous methods that utilize exponential decay for shape recognition, TSLTD employs linear time decay. This approach creates a predictable intensity gradient where pixel brightness reflects the recency of an event. This gradient effectively encodes both the direction and magnitude of motion into a 2D synchronous frame.

The proposed pipeline: asynchronous retinal events are encoded into a synchronous sequence of five TSLTD frames for processing by the RMRNet.

RMRNet Architecture

The Retinal Motion Regression Network (RMRNet) is a lightweight architecture optimized for real-time performance. It consists of CNN layers for spatial feature extraction and an LSTM module to capture temporal dynamics. By processing a sequence of five consecutive TSLTD frames, the LSTM allows the network to estimate accumulated motion accurately. The network ultimately branches into five regression heads to predict horizontal/vertical displacement ($d_x, d_y$), rotation ($\theta$), and horizontal/vertical scaling ($s_x, s_y$).

Experimental Evaluation

The model was rigorously tested on two datasets: the standard Event Camera Dataset (ECD) and the more challenging Extreme Event Dataset (EED). The EED features high-speed drone motion, low-light conditions, and significant background clutter designed to push sensors to their limits.

A key insight from the development was the superiority of linear decay over exponential decay for motion estimation. While exponential decay is useful for static object recognition, the linear gradient in TSLTD is more easily mapped to velocity by a CNN. Furthermore, the researchers determined that a sequence of exactly five frames provided the optimal balance between motion context and computational latency.

Key Results

The RMRNet demonstrated significant improvements over state-of-the-art (SOTA) frame-based trackers:

  • High-Speed Performance: On the EED fast_drone sequence, RMRNet achieved an Average Overlap Rate (AOR) of 0.892, outperforming trackers like SiamRPN++ (0.717) and ATOM (0.763), which were hampered by motion blur.
  • Low-Light Robustness: The system maintained high tracking accuracy in near-total darkness, conditions where conventional cameras failed to produce usable images.
  • Computational Efficiency: The entire pipeline, including TSLTD generation and inference, runs at approximately 38.57 ms per object pair, making it suitable for real-time applications.

Conclusion and Future Work

While RMRNet excels in high-speed and low-light scenarios, it faces challenges with total occlusions and dense background noise. Because the network relies on a continuous event stream, interruptions in the data can disrupt motion estimation. Future research will focus on incorporating re-identification logic to handle long-term occlusions and improving noise filtering to better isolate target objects in cluttered environments.

Summary updated

Abstract

Event cameras, which are asynchronous bio-inspired vision sensors, have shown great potential in computer vision and artificial intelligence. However, the application of event cameras to object-level motion estimation or tracking is still in its infancy. The main idea behind this work is to propose a novel deep neural network to learn and regress a parametric object-level motion/transform model for event-based object tracking. To achieve this goal, we propose a synchronous Time-Surface with Linear Time Decay (TSLTD) representation, which effectively encodes the spatio-temporal information of asynchronous retinal events into TSLTD frames with clear motion patterns. We feed the sequence of TSLTD frames to a novel Retinal Motion Regression Network (RMRNet) to perform an end-to-end 5-DoF object motion regression. Our method is compared with state-of-the-art object tracking methods, that are based on conventional cameras or event cameras. The experimental results show the superiority of our method in handling various challenging environments such as fast motion and low illumination conditions.
BibTeX
@inproceedings{chen2020end,
  author = {Haosheng Chen and David Suter and Qiangqiang Wu and Hanzi Wang},
  title = {{End-to-End Learning of Object Motion Estimation from Retinal Events for Event-Based Object Tracking}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.1609/aaai.v34i07.6625},
}
Figure