Abstract
Fusing Events and RGB images for object detection leverages the robustness of Event cameras in adverse environments and the rich semantic information provided by RGB cameras. However, two critical mismatches: low-latency Events vs. high-latency RGB frames; temporally sparse labels in training vs. continuous flow in inference, significantly hinder the high-frequency fusion-based object detection. To address these challenges, we propose the Frequency-Adaptive Low-Latency Object Detector (FAOD). FAOD aligns low-frequency RGB frames with high-frequency Events through an Align Module, which reinforces cross-modal style and spatial proximity to address the Event-RGB Mismatch. We further propose a training strategy, Time Shift, which enforces the module to align the prediction from temporally shifted Event-RGB pairs and their original representation, that is, consistent with Event-aligned annotations. This strategy enables the network to use high-frequency Event data as the primary reference while treating low-frequency RGB images as supplementary information, retaining the low-latency nature of the Event stream toward high-frequency detection. Furthermore, we observe that these corrected Event-RGB pairs demonstrate better generalization from low training frequency to higher inference frequencies compared to using Event data alone. Extensive experiments on the PKU-DAVIS-SOD and DSEC-Detection datasets demonstrate that our FAOD achieves SOTA performance. Specifically, in the PKU-DAVIS-SOD Dataset, FAOD achieves 9.8 points improvement in terms of the mAP in fully paired Event-RGB data with only a quarter of the parameters compared to SODFormer, and even maintains robust performance (only a 3 points drop in mAP) under 80x Event-RGB frequency mismatch.
Introduction: The Challenge of Multi-Modal Fusion
Object detection in high-speed or high-dynamic-range environments is a classic computer vision challenge. Standard RGB cameras provide rich semantic detail but suffer from motion blur and low frame rates (typically 30–60 fps). Event cameras, which only record per-pixel intensity changes, offer sub-millisecond latency and massive dynamic range but lack the texture and color needed for robust classification.
Fusing these two sensors seems like an obvious win, but two "critical mismatches" break existing models:
- Event-RGB Mismatch: Events are essentially a continuous stream (up to 10,000+ fps), while RGB frames are sparse. This leads to temporal and spatial misalignment when trying to fuse a "lagging" RGB frame with "current" event data.
- Train-Infer Mismatch: Most datasets provide labels at low frequencies (e.g., 25Hz). When a model trained on 25Hz data is asked to perform inference at 200Hz, the internal "memory" of the network often decays or fails to generalize, leading to a sharp drop in performance.
Proposed Method: FAOD
The proposed Frequency-Adaptive Low-Latency Object Detector (FAOD) utilizes three key components to bridge these gaps:
The Align Module
Instead of simply concatenating features, this module uses Adaptive Instance Normalization (AdaIN) to match the "style" of RGB features to event features. It then employs Deformable Convolutions to spatially warp the RGB features, effectively "moving" the old RGB frame forward in time to align with the current events.
Time Shift Training Strategy
To force the Align Module to learn, the authors randomly shift RGB frames forward in time during training. This simulates the lag found in real-world high-speed scenarios. It teaches the network to treat high-frequency events as the primary timing reference while using the corrected RGB data as supplementary semantic information.
Efficient Architecture
FAOD achieves better results than heavy transformer-based models (like SODFormer) while using only a quarter of the parameters (20.3M). It uses shallow feature fusion at the first stage and propagates them through a lightweight backbone (CSPDarkNet + LSTM), keeping the model fast with a 13.6ms inference time.
Experimental Results
The model was evaluated on the PKU-DAVIS-SOD and DSEC-Detection datasets.
Superior Accuracy and Robustness
On the PKU-DAVIS-SOD dataset, FAOD achieved 30.5 mAP, a massive 9.8-point improvement over the previous state-of-the-art (SODFormer). Crucially, in tests with an 80× frequency mismatch, FAOD saw only a 3-point drop in mAP, whereas competing methods often collapsed. This demonstrates that by providing the memory network with "corrected" RGB-Event pairs, the reliance on long-term temporal memory is reduced, allowing the model to generalize to much higher inference frequencies.
Conclusion
FAOD successfully addresses the temporal and spatial misalignment between events and frames. By treating events as the anchor for timing and using the Time Shift strategy, it maintains high accuracy even when inference speeds are significantly faster than the training data. Future work could involve exploring more complex "Time Shift" patterns or integrating advanced State Space Models (SSMs) for further temporal reasoning.
Summary updated