Introduction
Traditional computer vision relies on frame-based cameras that capture images at fixed intervals. While effective for static scenes, these sensors often fail in dynamic environments due to motion blur during high-speed movement or dynamic range limitations in high-contrast lighting. Event cameras, such as the DAVIS sensor, offer a bio-inspired alternative by reporting pixel-level brightness changes asynchronously as a stream of "retinal events."
Despite their advantages—high temporal resolution and high dynamic range—event cameras produce sparse, irregular data that is incompatible with standard convolutional neural networks (CNNs). This paper presents an end-to-end deep learning framework designed to estimate 5-degree-of-freedom (5-DoF) object motion directly from these events, enabling robust tracking in extreme conditions.
Methodology: TSLTD and RMRNet
The proposed system bridges the gap between asynchronous events and synchronous deep learning through a two-stage process: encoding and regression.
TSLTD Representation
To transform raw events into a format suitable for CNNs, the authors introduce TSLTD (Time-Surface with Linear Time Decay). Unlike previous methods that utilize exponential decay for shape recognition, TSLTD employs linear time decay. This approach creates a predictable intensity gradient where pixel brightness reflects the recency of an event. This gradient effectively encodes both the direction and magnitude of motion into a 2D synchronous frame.
RMRNet Architecture
The Retinal Motion Regression Network (RMRNet) is a lightweight architecture optimized for real-time performance. It consists of CNN layers for spatial feature extraction and an LSTM module to capture temporal dynamics. By processing a sequence of five consecutive TSLTD frames, the LSTM allows the network to estimate accumulated motion accurately. The network ultimately branches into five regression heads to predict horizontal/vertical displacement ($d_x, d_y$), rotation ($\theta$), and horizontal/vertical scaling ($s_x, s_y$).
Experimental Evaluation
The model was rigorously tested on two datasets: the standard Event Camera Dataset (ECD) and the more challenging Extreme Event Dataset (EED). The EED features high-speed drone motion, low-light conditions, and significant background clutter designed to push sensors to their limits.
A key insight from the development was the superiority of linear decay over exponential decay for motion estimation. While exponential decay is useful for static object recognition, the linear gradient in TSLTD is more easily mapped to velocity by a CNN. Furthermore, the researchers determined that a sequence of exactly five frames provided the optimal balance between motion context and computational latency.
Key Results
The RMRNet demonstrated significant improvements over state-of-the-art (SOTA) frame-based trackers:
- High-Speed Performance: On the EED fast_drone sequence, RMRNet achieved an Average Overlap Rate (AOR) of 0.892, outperforming trackers like SiamRPN++ (0.717) and ATOM (0.763), which were hampered by motion blur.
- Low-Light Robustness: The system maintained high tracking accuracy in near-total darkness, conditions where conventional cameras failed to produce usable images.
- Computational Efficiency: The entire pipeline, including TSLTD generation and inference, runs at approximately 38.57 ms per object pair, making it suitable for real-time applications.
Conclusion and Future Work
While RMRNet excels in high-speed and low-light scenarios, it faces challenges with total occlusions and dense background noise. Because the network relies on a continuous event stream, interruptions in the data can disrupt motion estimation. Future research will focus on incorporating re-identification logic to handle long-term occlusions and improving noise filtering to better isolate target objects in cluttered environments.
Summary updated