Event-based Moving Object Detection and Tracking

Anton Mitrokhin, Cornelia Fermüller, Chethan Parameshwara, Yiannis Aloimonos

Introduction

Traditional frame-based computer vision often struggles in high-speed scenarios due to motion blur, or in high dynamic range environments where lighting is extreme. Event-based sensors, such as the Dynamic Vision Sensor (DVS), address these issues by reporting asynchronous pixel-level intensity changes with microsecond resolution. However, a significant challenge remains: detecting independent motion from a moving platform. When the camera moves, every pixel generates events, making it difficult to distinguish between background ego-motion and moving objects.

Methodology

The proposed framework treats the event stream as a 3D point cloud and applies a motion compensation pipeline to separate global camera motion from independent object motion.

Motion Model and Time-Image

The authors approximate camera motion using a 4-parameter model (x-shift, y-shift, radial expansion, and 2D rotation). A key innovation is the Time-Image ($\mathcal{T}$), a 2D grid where each pixel stores the average timestamp of events warped to that location. Unlike raw event counts, the Time-Image provides smooth gradients suitable for optimization.

Two-Stage Optimization

The pipeline employs a two-stage process:

  1. Coarse Alignment (Algorithm 1): Minimizes spatial gradients of the Time-Image to reduce temporal "blur."
  2. Fine Refinement (Algorithm 2): Maximizes event density in a standard event-count image to achieve sharpness.
Figure 6: The proposed motion compensation and tracking pipeline, illustrating the two-stage optimization process and object extraction.

Once the background is stabilized, any remaining "blurry" event clusters are identified as independently moving objects and tracked using a Kalman Filter.

Experimental Results

The framework was evaluated on the Extreme Event Dataset (EED), which includes challenging scenarios like fast-moving drones and strobe-lit environments.

Figure 2: Comparison of raw event data (a, b) and motion-compensated results (c, d). The alignment significantly sharpens the scene, allowing for the isolation of moving objects.

Performance and Robustness

The system achieved detection success rates between 84.5% and 92.8%. Notably, it remained robust under extreme conditions, such as 1-2 Hz flashing strobe lights, where traditional cameras would fail. In terms of speed, a GPU implementation (NVIDIA Titan X) enabled processing at tens of thousands of frames per second equivalent, a 1000x-2300x speedup over CPU-based optimization.

Conclusion

This research demonstrates a robust method for real-time object tracking using event cameras by framing ego-motion compensation as a 3D alignment problem. While the system struggles with zero relative motion or extreme lack of background texture, it provides a high-performance solution for high-speed robotic navigation and surveillance.

Summary updated

Abstract

Event-based vision sensors, such as the Dynamic Vision Sensor (DVS), are ideally suited for real-time motion analysis. The unique properties encompassed in the readings of such sensors provide high temporal resolution, superior sensitivity to light and low latency. These properties provide the grounds to estimate motion extremely reliably in the most sophisticated scenarios but they come at a price - modern event-based vision sensors have extremely low resolution and produce a lot of noise. Moreover, the asynchronous nature of the event stream calls for novel algorithms. This paper presents a new, efficient approach to object tracking with asynchronous cameras. We present a novel event stream representation which enables us to utilize information about the dynamic (temporal) component of the event stream, and not only the spatial component, at every moment of time. This is done by approximating the 3D geometry of the event stream with a parametric model; as a result, the algorithm is capable of producing the motion-compensated event stream (effectively approximating egomotion), and without using any form of external sensors in extremely low-light and noisy conditions without any form of feature tracking or explicit optical flow computation. We demonstrate our framework on the task of independent motion detection and tracking, where we use the temporal model inconsistencies to locate differently moving objects in challenging situations of very fast motion.
BibTeX
@inproceedings{mitrokhin2018event,
  author = {Anton Mitrokhin and Cornelia Fermüller and Chethan Parameshwara and Yiannis Aloimonos},
  title = {{Event-based Moving Object Detection and Tracking}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2018},
  doi = {10.1109/iros.2018.8593805},
}
Figure