Detecting Every Object From Events

Haitian Zhang, Chang Xu, Xinya Wang, Bingde Liu, Guang Hua, Lei Yu, Wen Yang

TL;DR

DEOE (Detecting Every Object from Events) is a novel framework designed to solve the "Class-Agnostic Object Detection" (CAOD) problem using event cameras. By leveraging the sub-millisecond temporal resolution of event streams and a "disentangled" architecture, DEOE can identify and localize objects it has never seen before—even in extreme lighting or high-speed scenarios where traditional cameras fail.

What Problem Does This Paper Address?

Standard object detection models operate under a "closed-set" assumption: they can only detect categories they were explicitly trained on (e.g., "car," "pedestrian"). In the real world—especially for autonomous driving—this is a safety risk. A vehicle needs to detect any obstacle in its path, whether it's a known category or an "unknown" object like a fallen tree or a rare animal. This is the goal of Class-Agnostic Object Detection (CAOD).

While CAOD has been explored with traditional RGB cameras, these sensors suffer from motion blur and limited dynamic range. Event cameras are a promising alternative because they only record changes in brightness at the pixel level with microsecond latency. However, existing CAOD methods often treat unannotated (unknown) objects as "background" during training, effectively teaching the model to ignore them. DEOE aims to bridge this gap by using the unique properties of event data to discover these hidden objects.

Methods

The authors propose a framework built on the Recurrent Vision Transformer (RVT) backbone. The core innovation lies in how the model identifies "potential" objects without explicit labels:

  1. Spatio-temporal Consistency:
    • Spatial: The model uses a Dual Regressor Head with dropout layers to introduce randomness. If two independent predictions for the same area result in highly overlapping boxes (High Spatial IoU), the model assumes it has found a real object rather than random noise.
  2. Temporal: Because event cameras capture data nearly continuously, an object’s position changes negligibly between consecutive millisecond intervals. The model calculates a Temporal IoU; high overlap across time steps strongly indicates a persistent physical object.
  3. Disentangled Objectness Head:
    • Instead of one classifier, DEOE uses two branches. The Positive-Negative branch focuses on distinguishing known foreground objects from the background.
  4. The Positive-Only branch is designed to "excavate" unknown objects. It treats identified potential samples as "soft positives," preventing the model from penalizing the detection of unannotated objects.
  5. Sample Screening: A scoring function combines objectness confidence, spatial consistency, and temporal consistency to select the best "potential" samples to reinforce the training signal.

Datasets

  • 1 MEGAPIXEL Automotive Detection Dataset (1 Mpx): The primary benchmark. The authors used a 4-hour high-confidence subset containing roughly 7.5 million annotated boxes across 7 categories (pedestrians, trucks, etc.).
  • DSEC-Detection: Used for cross-dataset validation to test how well the model generalizes to entirely new environments.
  • Evaluation Settings: The authors used "5+2" and "4+3" splits, where 5 (or 4) classes are "known" during training, and the remaining 2 (or 3) are treated as "unknown" during testing.

Key Insights and Ideas

  • Randomness as a Filter: By using dropout in the regression heads, the authors turned a common regularization technique into a diagnostic tool. If a prediction is "stable" despite the injected randomness, it’s likely a real object.
  • The "Soft Positive" Strategy: The biggest hurdle in CAOD is "missing labels." By treating high-confidence unannotated regions as soft positives rather than hard negatives, the model maintains the flexibility to detect novel shapes.
  • Events are Superior for Motion: The paper demonstrates that the high temporal resolution of events isn't just a speed advantage; it's a structural advantage for object discovery. The "Temporal IoU" metric is far more reliable at 1000Hz (event scale) than at 30Hz (standard video scale).

Key Results

  • Superior Recall: In the "5+2" setting on the 1 Mpx dataset, DEOE achieved an Average Recall (AR300) of 35.28% for unknown classes, significantly outperforming the baseline CA-RVT (25.74%).
  • Beating the "Oracle": Remarkably, DEOE outperformed the "Oracle" model (a model trained with all labels) on several metrics. This suggests that DEOE’s consistency-based discovery can find objects that even human annotators (or the pseudo-labeling algorithms used to create the dataset) might have missed.
  • Real-time Performance: Despite the dual-head architecture, DEOE maintains high efficiency, achieving over 100 FPS on an RTX 4090, making it viable for real-time robotics.
  • Robustness: Qualitative results show DEOE successfully detecting "two-wheelers" and "trucks" in pitch-black conditions where RGB-based models saw nothing.

Limitations and Future Work

  • Semantic Confusion: Because the model lacks color information, it occasionally triggers "false alarms" on background textures that have object-like contours (e.g., certain road markings or shadows).
  • Small Object Sensitivity: Like many detectors, DEOE struggles with very small, distant objects where the event stream is too sparse to establish strong spatio-temporal consistency.
  • Future Directions: The authors suggest exploring multi-modal fusion (combining events with sparse RGB frames) to reduce false alarms while maintaining high-speed detection capabilities.

Summary updated

Abstract

Object detection is critical in autonomous driving, and it is more practical yet challenging to localize objects of unknown categories: an endeavour known as Class-Agnostic Object Detection (CAOD). Existing studies on CAOD predominantly rely on ordinary cameras, but these frame-based sensors usually have high latency and limited dynamic range, leading to safety risks in real-world scenarios. In this study, we turn to a new modality enabled by the so-called event camera, featured by its sub-millisecond latency and high dynamic range, for robust CAOD. We propose Detecting Every Object in Events (DEOE), an approach tailored for achieving high-speed, class-agnostic open-world object detection in event-based vision. Built upon the fast event-based backbone: recurrent vision transformer, we jointly consider the spatial and temporal consistencies to identify potential objects. The discovered potential objects are assimilated as soft positive samples to avoid being suppressed as background. Moreover, we introduce a disentangled objectness head to separate the foreground-background classification and novel object discovery tasks, enhancing the model's generalization in localizing novel objects while maintaining a strong ability to filter out the background. Extensive experiments confirm the superiority of our proposed DEOE in comparison with three strong baseline methods that integrate the state-of-the-art event-based object detector with advancements in RGB-based CAOD. Our code is available at https://github.com/Hatins/DEOE.
BibTeX
@article{zhang2025detecting,
  author = {Haitian Zhang and Chang Xu and Xinya Wang and Bingde Liu and Guang Hua and Lei Yu and Wen Yang},
  title = {{Detecting Every Object From Events}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2025},
  doi = {10.1109/tpami.2025.3565102},
}

Loading related content...

Figure