TL;DR
DEOE (Detecting Every Object from Events) is a novel framework designed to solve the "Class-Agnostic Object Detection" (CAOD) problem using event cameras. By leveraging the sub-millisecond temporal resolution of event streams and a "disentangled" architecture, DEOE can identify and localize objects it has never seen before—even in extreme lighting or high-speed scenarios where traditional cameras fail.
What Problem Does This Paper Address?
Standard object detection models operate under a "closed-set" assumption: they can only detect categories they were explicitly trained on (e.g., "car," "pedestrian"). In the real world—especially for autonomous driving—this is a safety risk. A vehicle needs to detect any obstacle in its path, whether it's a known category or an "unknown" object like a fallen tree or a rare animal. This is the goal of Class-Agnostic Object Detection (CAOD).
While CAOD has been explored with traditional RGB cameras, these sensors suffer from motion blur and limited dynamic range. Event cameras are a promising alternative because they only record changes in brightness at the pixel level with microsecond latency. However, existing CAOD methods often treat unannotated (unknown) objects as "background" during training, effectively teaching the model to ignore them. DEOE aims to bridge this gap by using the unique properties of event data to discover these hidden objects.
Methods
The authors propose a framework built on the Recurrent Vision Transformer (RVT) backbone. The core innovation lies in how the model identifies "potential" objects without explicit labels:
- Spatio-temporal Consistency:
- Spatial: The model uses a Dual Regressor Head with dropout layers to introduce randomness. If two independent predictions for the same area result in highly overlapping boxes (High Spatial IoU), the model assumes it has found a real object rather than random noise.
- Temporal: Because event cameras capture data nearly continuously, an object’s position changes negligibly between consecutive millisecond intervals. The model calculates a Temporal IoU; high overlap across time steps strongly indicates a persistent physical object.
- Disentangled Objectness Head:
- Instead of one classifier, DEOE uses two branches. The Positive-Negative branch focuses on distinguishing known foreground objects from the background.
- The Positive-Only branch is designed to "excavate" unknown objects. It treats identified potential samples as "soft positives," preventing the model from penalizing the detection of unannotated objects.
- Sample Screening: A scoring function combines objectness confidence, spatial consistency, and temporal consistency to select the best "potential" samples to reinforce the training signal.
Datasets
- 1 MEGAPIXEL Automotive Detection Dataset (1 Mpx): The primary benchmark. The authors used a 4-hour high-confidence subset containing roughly 7.5 million annotated boxes across 7 categories (pedestrians, trucks, etc.).
- DSEC-Detection: Used for cross-dataset validation to test how well the model generalizes to entirely new environments.
- Evaluation Settings: The authors used "5+2" and "4+3" splits, where 5 (or 4) classes are "known" during training, and the remaining 2 (or 3) are treated as "unknown" during testing.
Key Insights and Ideas
- Randomness as a Filter: By using dropout in the regression heads, the authors turned a common regularization technique into a diagnostic tool. If a prediction is "stable" despite the injected randomness, it’s likely a real object.
- The "Soft Positive" Strategy: The biggest hurdle in CAOD is "missing labels." By treating high-confidence unannotated regions as soft positives rather than hard negatives, the model maintains the flexibility to detect novel shapes.
- Events are Superior for Motion: The paper demonstrates that the high temporal resolution of events isn't just a speed advantage; it's a structural advantage for object discovery. The "Temporal IoU" metric is far more reliable at 1000Hz (event scale) than at 30Hz (standard video scale).
Key Results
- Superior Recall: In the "5+2" setting on the 1 Mpx dataset, DEOE achieved an Average Recall (AR300) of 35.28% for unknown classes, significantly outperforming the baseline CA-RVT (25.74%).
- Beating the "Oracle": Remarkably, DEOE outperformed the "Oracle" model (a model trained with all labels) on several metrics. This suggests that DEOE’s consistency-based discovery can find objects that even human annotators (or the pseudo-labeling algorithms used to create the dataset) might have missed.
- Real-time Performance: Despite the dual-head architecture, DEOE maintains high efficiency, achieving over 100 FPS on an RTX 4090, making it viable for real-time robotics.
- Robustness: Qualitative results show DEOE successfully detecting "two-wheelers" and "trucks" in pitch-black conditions where RGB-based models saw nothing.
Limitations and Future Work
- Semantic Confusion: Because the model lacks color information, it occasionally triggers "false alarms" on background textures that have object-like contours (e.g., certain road markings or shadows).
- Small Object Sensitivity: Like many detectors, DEOE struggles with very small, distant objects where the event stream is too sparse to establish strong spatio-temporal consistency.
- Future Directions: The authors suggest exploring multi-modal fusion (combining events with sparse RGB frames) to reduce false alarms while maintaining high-speed detection capabilities.
Summary updated