Introduction
Standard RGB cameras are the primary sensors for computer vision, yet they possess a significant "glass jaw": performance degrades rapidly under extreme lighting conditions—such as underexposure in low light or overexposure from glare—and high-speed motion blur. Event cameras offer a compelling alternative, reporting per-pixel brightness changes with microsecond latency and high dynamic range. However, integrating these sensors effectively has been hindered by existing datasets that suffer from low resolution, spatial misalignment, and a lack of diverse "edge case" scenarios.
Methodology
To overcome the challenge of spatial misalignment, the researchers developed a custom Coaxial Imaging System. By utilizing a beam splitter and a coaxial cube, the system ensures that the RGB camera and the Event camera share the exact same optical path, achieving true pixel-level alignment. A hardware signal generator provides a master clock to synchronize both sensors to the microsecond.
Hybrid Annotation Strategy
Labeling objects in pitch-black or blurry RGB frames is nearly impossible for human annotators. To establish accurate ground truth, the team employed an AI reconstruction algorithm (NER-Net) to transform raw event streams into high-clarity grayscale images. Annotators then drew bounding boxes on these reconstructions, which were subsequently mapped back to the pixel-aligned RGB frames.
The PEOD Dataset
The Pixel-aligned Event-RGB Object Detection (PEOD) dataset represents a significant leap in benchmark quality:
- High Resolution: 1280 × 720 (720p), enabling detection of small, distant objects.
- Scale: Over 130 sequences containing 72,000 frames and 340,000 manual annotations across six classes (car, bus, truck, two-wheeler, three-wheeler, person).
- Challenging Conditions: 57.1% of the data is captured in "Illumination Challenge" environments, including night scenes, tunnels, and intense glare.
Experimental Results and Insights
The authors benchmarked 14 models, revealing critical insights into multi-modal fusion:
- The "Lazy Fusion" Problem: While fusion models like EOLO achieved the highest overall mAP (26.1), their performance plummeted in extreme conditions. In the toughest lighting, the top event-only model (SMamba, 23.2 mAP) significantly outperformed the top fusion model (EOLO, 11.5 mAP). This suggests that current fusion strategies are "lazy"—they over-rely on RGB textures and fail to ignore RGB "noise" when frames become unreliable.
- Event-Only Strengths: SMamba, utilizing the Mamba architecture for sequence modeling, proved highly effective for event data, highlighting the importance of temporal modeling in this modality.
- RGB Vulnerability: The strongest RGB-only baseline, RT-DETR, saw its performance drop from 21.8 mAP overall to just 13.2 mAP in illumination-challenged subsets.
Conclusion and Future Work
PEOD establishes a new standard for high-resolution, pixel-aligned multi-modal detection. The benchmark reveals a massive opportunity for "reliability-aware" fusion research. Future work should focus on adaptive reweighting mechanisms that allow models to dynamically trust event data more heavily when the RGB signal-to-noise ratio drops. Additionally, the continuous nature of PEOD's sequences makes it a prime candidate for advancing multi-object tracking (MOT) in extreme environments.
Summary updated