Motion Robust High-Speed Light-Weighted Object Detection With Event Camera

Bingde Liu, Chang Xu, Wen Yang, Huai Yu, Lei Yu

TL;DR

This paper introduces a novel, lightweight object detection system for event cameras that addresses the challenges of high temporal resolution and asynchronous data. By proposing a new event representation (TAF) and a specialized detector (AED), the method achieves competitive accuracy with significantly reduced computational cost and improved robustness to object motion.

What Problem Does This Paper Address?

Traditional object detection methods struggle with the unique characteristics of event cameras, which generate sparse, asynchronous data streams with extremely high temporal resolution. Existing approaches often fail to fully leverage this high temporal resolution, relying on heavy models that are too slow for real-time applications or using fixed temporal windows that cannot effectively handle objects moving at different velocities relative to the camera. This leads to missed detections or inaccurate bounding boxes, especially for fast-moving objects or in scenarios with varying object speeds. The paper aims to bridge this gap by developing a motion-robust, high-speed, and lightweight object detection pipeline specifically designed for event camera data.

Methods

The proposed system comprises three main components:

  • Temporal Active Focus (TAF): This novel event stream representation method addresses the limitations of fixed temporal windows. TAF asynchronously samples event data at each spatial and polar position, creating dense tensor representations that are robust to object motion. It dynamically adjusts the time window length and temporal resolution based on event triggering frequency, allowing for better information aggregation for both slow and fast-moving objects.
  • Bifurcated Folding Module (BFM): This module is designed to extract rich temporal information encoded in the TAF tensor. It aggregates values at adjacent temporal positions, prioritizing recent information, and uses a slicing mechanism inspired by CSPNet to enhance temporal feature extraction. This module is applied point-wise at each spatial location.
  • Agile Event Detector (AED): This is a lightweight object detector built upon the YOLOX architecture. It uses a modified Darknet21 backbone for faster processing and incorporates a Feature Pyramid Network (FPN) and a decoupled detection head. The AED is designed for high inference speed, making it suitable for real-time event camera applications.

Additionally, the paper proposes a simple yet effective data augmentation strategy involving random flipping and random cropping/resizing to improve the generalization capability of the detector.

Datasets

The authors evaluated their method on two typical real-scene event camera object detection datasets:

  • Prophesee GEN1 Automotive Detection Dataset (GEN1 Dataset): A complete dataset for automotive object detection.
  • Prophesee 1 MEGAPIXEL Automotive Detection Dataset (1 MEGAPIXEL Dataset (Subset)): A subset of a larger dataset with partial annotations, used to evaluate performance under more constrained conditions.

The motion speed of objects in these datasets was analyzed by computing optical flow and classifying them into five motion levels.

Key Insights and Ideas

  • Asynchronous Sampling is Crucial: The paper highlights that fixed temporal windows are a significant bottleneck for event-based object detection, as they cannot adapt to varying object speeds. TAF's asynchronous sampling approach is key to overcoming this.
  • Temporal Information is Rich but Challenging: Event camera data is rich in temporal information, but effectively encoding it requires specialized modules like BFM, which can model local temporal connections and prioritize recent events.
  • Lightweight Architectures are Necessary: High temporal resolution demands fast processing. The AED detector, by being lightweight, enables real-time performance that is not achievable with heavier, traditional detectors.
  • Motion Robustness is Achievable: By carefully designing the event representation and detector, the method demonstrates strong performance across different object motion levels, a critical aspect for safety-critical applications like autonomous driving.

Key Results

  • Competitive Accuracy: The proposed method achieves competitive accuracy compared to state-of-the-art methods on both datasets. For instance, it achieves 0.454 mAP on the GEN1 Dataset and 0.344 mAP on the 1 MEGAPIXEL Dataset (Subset).
  • Significant Speedup: The AED detector, combined with TAF, offers substantial improvements in inference speed. On the GEN1 Dataset, it achieves 74.6 FPS, and on the 1 MEGAPIXEL Dataset (Subset), it reaches 61.8 FPS. This is a significant improvement over many existing methods.
  • Reduced Parameters: The method is considerably more lightweight, requiring fewer parameters than many comparable state-of-the-art approaches. For example, it uses only 14.8M parameters, a fraction of what some other methods require.
  • Motion Robustness: The method demonstrates high detection accuracy across all five defined motion levels, indicating its robustness to objects moving at different speeds relative to the camera.

Limitations and Future Work

  • Hyper-parameter Sensitivity (for other methods): The paper notes that traditional event representation methods (Event Volume, Event Count Image, Surface of Active Events) are sensitive to hyper-parameter choices, requiring different settings for low-motion vs. high-motion objects. TAF aims to mitigate this.
  • Computational Cost of BFM: While BFM improves accuracy, it slightly increases the running time due to feature extraction. The paper explores queue depth (K) as a hyper-parameter for BFM, suggesting further optimization might be possible.
  • Future Work: The authors suggest that further research could explore more advanced temporal aggregation techniques and potentially integrate TAF with other sensor modalities for enhanced perception. The generalizability of TAF to other event-based tasks beyond object detection is also an area for future exploration.

Summary updated

Abstract

The event camera asynchronously produces the event stream with a high temporal resolution, discarding redundant visual information and bringing new possibilities for moving object detection. Nevertheless, the existing object detectors cannot make the most of the spatial-temporal asynchronous nature and high temporal resolution of the event stream. For one thing, existing methods fail to consider objects with different velocities relative to the event camera’s motion, resulting from the global synchronized time window with the whole spatial slice. For another, most of the existing methods rely on heavy models and boost the detection performance with low frame rates, failing to utilize the high temporal resolution characteristic of the event stream. In this work, we propose a motion robust and high-speed detection pipeline which better leverages the event data. First, we design an event stream representation called temporal active focus (TAF), which efficiently utilizes the spatial-temporal asynchronous event stream, constructing event tensors robust to object motions. Then, we propose a module called the bifurcated folding module (BFM), which encodes the rich temporal information in the TAF tensor at the input layer of the detector. Following this, we design a high-speed lightweight detector called agile event detector (AED) plus a simple but effective data augmentation method, to enhance the detection accuracy and reduce the model’s parameter. Experiments on two typical real-scene event camera object detection datasets show that our method is competitive in terms of accuracy, efficiency, and the number of parameters. By classifying objects into multiple motion levels based on the optical flow density metric, we further illustrated the robustness of our method for objects with different velocities relative to the camera. The codes and trained models are available at <uri xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">https://github.com/HarmoniaLeo/FRLW-EvD</uri> .
BibTeX
@article{liu2023motion,
  author = {Bingde Liu and Chang Xu and Wen Yang and Huai Yu and Lei Yu},
  title = {{Motion Robust High-Speed Light-Weighted Object Detection With Event Camera}},
  journal = {IEEE Trans. Instrumentation \& Measurement},
  year = {2023},
  doi = {10.1109/tim.2023.3269780},
}
Figure