Detecting Every Object from Events

Haitian Zhang, Chang Xu, Xinya Wang, Bingde Liu, Guang Hua, Lei Yu, Wen Yang

Abstract

Object detection is critical in autonomous driving, and it is more practical yet challenging to localize objects of unknown categories: an endeavour known as Class-Agnostic Object Detection (CAOD). Existing studies on CAOD predominantly rely on ordinary cameras, but these frame-based sensors usually have high latency and limited dynamic range, leading to safety risks in real-world scenarios. In this study, we turn to a new modality enabled by the so-called event camera, featured by its sub-millisecond latency and high dynamic range, for robust CAOD. We propose Detecting Every Object in Events (DEOE), an approach tailored for achieving high-speed, class-agnostic open-world object detection in event-based vision. Built upon the fast event-based backbone: recurrent vision transformer, we jointly consider the spatial and temporal consistencies to identify potential objects. The discovered potential objects are assimilated as soft positive samples to avoid being suppressed as background. Moreover, we introduce a disentangled objectness head to separate the foreground-background classification and novel object discovery tasks, enhancing the model's generalization in localizing novel objects while maintaining a strong ability to filter out the background. Extensive experiments confirm the superiority of our proposed DEOE in comparison with three strong baseline methods. Our code is available at https://github.com/Hatins/DEOE.

Introduction

Standard object detection models operate under a "closed-set" assumption, meaning they can only detect categories they were explicitly trained on (e.g., "car," "pedestrian"). In real-world autonomous driving, this poses a significant safety risk, as vehicles must detect any obstacle, including unknown objects like fallen trees or rare animals. This challenge is addressed by Class-Agnostic Object Detection (CAOD).

While CAOD has been explored with traditional RGB cameras, these sensors suffer from motion blur and limited dynamic range. Event cameras offer a promising alternative by recording brightness changes at the pixel level with microsecond latency. However, existing CAOD methods often treat unannotated (unknown) objects as background during training. DEOE (Detecting Every Object from Events) bridges this gap by leveraging the unique properties of event data to discover hidden objects.

Overview of the DEOE framework showing the discovery of potential objects using spatial and temporal consistencies.

Methodology

DEOE is built upon the Recurrent Vision Transformer (RVT) backbone. The framework identifies "potential" objects without explicit labels through two primary mechanisms:

Spatio-temporal Consistency

  1. Spatial Consistency: The model employs a Dual Regressor Head with dropout layers to introduce randomness. If two independent predictions for the same area result in highly overlapping boxes (High Spatial IoU), the model assumes it has found a real object rather than random noise.
  2. Temporal Consistency: Due to the high temporal resolution of event cameras, an object’s position changes minimally between consecutive millisecond intervals. DEOE calculates a Temporal IoU; high overlap across time steps indicates a persistent physical object.

Disentangled Objectness Head

To prevent unknown objects from being suppressed as background, DEOE uses a dual-branch architecture:

  • Positive-Negative Branch: Focuses on distinguishing known foreground objects from the background.
  • Positive-Only Branch: Designed to "excavate" unknown objects. It treats identified potential samples as "soft positives," allowing the model to maintain flexibility in detecting novel shapes.
Detailed architecture of DEOE, illustrating the recurrent pipeline and the disentangled head structure.

Experiments and Results

The model was evaluated on the 1 MEGAPIXEL Automotive Detection Dataset (1 Mpx) and the DSEC-Detection dataset for cross-dataset validation. Using "5+2" and "4+3" class splits (where some classes are withheld during training), DEOE demonstrated significant improvements over baselines.

  • Superior Recall: In the "5+2" setting on the 1 Mpx dataset, DEOE achieved an Average Recall (AR300) of 35.28% for unknown classes, significantly outperforming the baseline CA-RVT (25.74%).
  • Real-time Performance: Despite its complexity, DEOE maintains high efficiency, achieving over 100 FPS on an RTX 4090.
  • Robustness: Qualitative results show DEOE successfully detecting objects in pitch-black conditions where RGB-based models fail entirely.

Conclusion

DEOE demonstrates that the high temporal resolution of event cameras is a structural advantage for object discovery. By treating high-confidence unannotated regions as soft positives and utilizing spatio-temporal consistency, the model achieves state-of-the-art performance in class-agnostic detection. Future work will explore multi-modal fusion to further reduce false alarms from background textures while maintaining high-speed capabilities.

Summary updated

Abstract

Object detection is critical in autonomous driving, and it is more practical yet challenging to localize objects of unknown categories: an endeavour known as Class-Agnostic Object Detection (CAOD). Existing studies on CAOD predominantly rely on ordinary cameras, but these frame-based sensors usually have high latency and limited dynamic range, leading to safety risks in real-world scenarios. In this study, we turn to a new modality enabled by the so-called event camera, featured by its sub-millisecond latency and high dynamic range, for robust CAOD. We propose Detecting Every Object in Events (DEOE), an approach tailored for achieving high-speed, class-agnostic open-world object detection in event-based vision. Built upon the fast event-based backbone: recurrent vision transformer, we jointly consider the spatial and temporal consistencies to identify potential objects. The discovered potential objects are assimilated as soft positive samples to avoid being suppressed as background. Moreover, we introduce a disentangled objectness head to separate the foreground-background classification and novel object discovery tasks, enhancing the model's generalization in localizing novel objects while maintaining a strong ability to filter out the background. Extensive experiments confirm the superiority of our proposed DEOE in comparison with three strong baseline methods that integrate the state-of-the-art event-based object detector with advancements in RGB-based CAOD. Our code is available at https://github.com/Hatins/DEOE.
BibTeX
@article{zhang2025detecting,
  author = {Haitian Zhang and Chang Xu and Xinya Wang and Bingde Liu and Guang Hua and Lei Yu and Wen Yang},
  title = {{Detecting Every Object from Events}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3565102},
}
Figure