Abstract
Event cameras, which are asynchronous bio-inspired vision sensors, have shown great potential in computer vision and artificial intelligence. However, the application of event cameras to object-level motion estimation or tracking is still in its infancy. The main idea behind this work is to propose a novel deep neural network to learn and regress a parametric object-level motion/transform model for event-based object tracking. To achieve this goal, we propose a synchronous Time-Surface with Linear Time Decay (TSLTD) representation, which effectively encodes the spatio-temporal information of asynchronous retinal events into TSLTD frames with clear motion patterns. We feed the sequence of TSLTD frames to a novel Retinal Motion Regression Network (RMRNet) to perform an end-to-end 5-DoF object motion regression. Our method is compared with state-of-the-art object tracking methods, that are based on conventional cameras or event cameras. The experimental results show the superiority of our method in handling various challenging environments such as fast motion and low illumination conditions.
Introduction
Standard video cameras capture frames at fixed intervals, which often results in motion blur during high-speed movement and a loss of detail in high dynamic range (HDR) scenes. Event cameras, such as the DAVIS sensor, offer a bio-inspired alternative by reporting pixel-level brightness changes (events) asynchronously with microsecond precision. While these sensors excel in extreme conditions, their sparse and irregular data output is difficult for standard deep learning architectures to process. This work addresses this gap by introducing RMRNet, a framework that regresses 5-DoF object motion directly from a novel event representation.
Methodology
The proposed pipeline consists of two primary components: a motion-aware data representation and a specialized regression network.
TSLTD Representation
To bridge the gap between asynchronous events and convolutional neural networks (CNNs), we aggregate events into Time-Surface with Linear Time Decay (TSLTD) frames. Unlike traditional exponential decay surfaces, TSLTD uses linear decay to produce clear intensity gradients. These gradients serve as visual signatures of an object's velocity and direction, effectively encoding motion information into a format suitable for spatial feature extraction.
RMRNet Architecture
The Retinal Motion Regression Network (RMRNet) processes a sequence of five TSLTD patches. The architecture comprises:
- CNN Layers: Four convolutional layers extract spatial features from the motion patterns in the TSLTD frames.
- LSTM Module: A Long Short-Term Memory (LSTM) module integrates these features over time, capturing the temporal evolution of the object's movement.
- 5-DoF Regression: The network outputs five transformation parameters: horizontal displacement ($d_x$), vertical displacement ($d_y$), rotation ($\theta$), and scaling ($s_x, s_y$).
Experimental Evaluation
We evaluated RMRNet on the Event Camera Dataset (ECD) and the Extreme Event Dataset (EED), comparing it against state-of-the-art (SOTA) trackers including SiamFC, ECO, and ATOM.
Performance in Extreme Conditions
RMRNet demonstrated significant advantages in scenarios where traditional frame-based cameras fail:
- Fast Motion: On the fast_drone sequence, RMRNet achieved an Average Overlap Rate (AOR) of 0.892, whereas traditional trackers struggled with motion blur.
- Low Illumination: The sensor's high dynamic range allowed RMRNet to maintain high tracking accuracy in dark environments.
- Occlusions: In the EED occlusions sequence, RMRNet reached an AOR of 0.716, nearly doubling the performance of the next best competitor, ATOM (0.387).
Efficiency
Despite the inclusion of an LSTM, RMRNet remains computationally efficient. It processes an object pair in approximately 38.57 ms on a standard GTX 1080 GPU, supporting real-time tracking applications.
Conclusion and Future Work
RMRNet provides a robust, end-to-end solution for event-based object tracking, particularly in high-speed and low-light environments. A current limitation is the handling of dense dynamic clutter, as seen in the what_is_background sequence, where foreground objects disrupt motion patterns. Future research will focus on improving robustness to dynamic occlusions and extending the model to more complex 3D motion estimation.
Summary updated