TL;DR
This paper introduces a novel, lightweight object detection system for event cameras that addresses the challenges of high temporal resolution and asynchronous data. By proposing a new event representation (TAF) and a specialized detector (AED), the method achieves competitive accuracy with significantly reduced computational cost and improved robustness to object motion.
What Problem Does This Paper Address?
Traditional object detection methods struggle with the unique characteristics of event cameras, which generate sparse, asynchronous data streams with extremely high temporal resolution. Existing approaches often fail to fully leverage this high temporal resolution, relying on heavy models that are too slow for real-time applications or using fixed temporal windows that cannot effectively handle objects moving at different velocities relative to the camera. This leads to missed detections or inaccurate bounding boxes, especially for fast-moving objects or in scenarios with varying object speeds. The paper aims to bridge this gap by developing a motion-robust, high-speed, and lightweight object detection pipeline specifically designed for event camera data.
Methods
The proposed system comprises three main components:
- Temporal Active Focus (TAF): This novel event stream representation method addresses the limitations of fixed temporal windows. TAF asynchronously samples event data at each spatial and polar position, creating dense tensor representations that are robust to object motion. It dynamically adjusts the time window length and temporal resolution based on event triggering frequency, allowing for better information aggregation for both slow and fast-moving objects.
- Bifurcated Folding Module (BFM): This module is designed to extract rich temporal information encoded in the TAF tensor. It aggregates values at adjacent temporal positions, prioritizing recent information, and uses a slicing mechanism inspired by CSPNet to enhance temporal feature extraction. This module is applied point-wise at each spatial location.
- Agile Event Detector (AED): This is a lightweight object detector built upon the YOLOX architecture. It uses a modified Darknet21 backbone for faster processing and incorporates a Feature Pyramid Network (FPN) and a decoupled detection head. The AED is designed for high inference speed, making it suitable for real-time event camera applications.
Additionally, the paper proposes a simple yet effective data augmentation strategy involving random flipping and random cropping/resizing to improve the generalization capability of the detector.
Datasets
The authors evaluated their method on two typical real-scene event camera object detection datasets:
- Prophesee GEN1 Automotive Detection Dataset (GEN1 Dataset): A complete dataset for automotive object detection.
- Prophesee 1 MEGAPIXEL Automotive Detection Dataset (1 MEGAPIXEL Dataset (Subset)): A subset of a larger dataset with partial annotations, used to evaluate performance under more constrained conditions.
The motion speed of objects in these datasets was analyzed by computing optical flow and classifying them into five motion levels.
Key Insights and Ideas
- Asynchronous Sampling is Crucial: The paper highlights that fixed temporal windows are a significant bottleneck for event-based object detection, as they cannot adapt to varying object speeds. TAF's asynchronous sampling approach is key to overcoming this.
- Temporal Information is Rich but Challenging: Event camera data is rich in temporal information, but effectively encoding it requires specialized modules like BFM, which can model local temporal connections and prioritize recent events.
- Lightweight Architectures are Necessary: High temporal resolution demands fast processing. The AED detector, by being lightweight, enables real-time performance that is not achievable with heavier, traditional detectors.
- Motion Robustness is Achievable: By carefully designing the event representation and detector, the method demonstrates strong performance across different object motion levels, a critical aspect for safety-critical applications like autonomous driving.
Key Results
- Competitive Accuracy: The proposed method achieves competitive accuracy compared to state-of-the-art methods on both datasets. For instance, it achieves 0.454 mAP on the GEN1 Dataset and 0.344 mAP on the 1 MEGAPIXEL Dataset (Subset).
- Significant Speedup: The AED detector, combined with TAF, offers substantial improvements in inference speed. On the GEN1 Dataset, it achieves 74.6 FPS, and on the 1 MEGAPIXEL Dataset (Subset), it reaches 61.8 FPS. This is a significant improvement over many existing methods.
- Reduced Parameters: The method is considerably more lightweight, requiring fewer parameters than many comparable state-of-the-art approaches. For example, it uses only 14.8M parameters, a fraction of what some other methods require.
- Motion Robustness: The method demonstrates high detection accuracy across all five defined motion levels, indicating its robustness to objects moving at different speeds relative to the camera.
Limitations and Future Work
- Hyper-parameter Sensitivity (for other methods): The paper notes that traditional event representation methods (Event Volume, Event Count Image, Surface of Active Events) are sensitive to hyper-parameter choices, requiring different settings for low-motion vs. high-motion objects. TAF aims to mitigate this.
- Computational Cost of BFM: While BFM improves accuracy, it slightly increases the running time due to feature extraction. The paper explores queue depth (K) as a hyper-parameter for BFM, suggesting further optimization might be possible.
- Future Work: The authors suggest that further research could explore more advanced temporal aggregation techniques and potentially integrate TAF with other sensor modalities for enhanced perception. The generalizability of TAF to other event-based tasks beyond object detection is also an area for future exploration.
Summary updated