PEOD: A Pixel-Aligned Event-RGB Benchmark for Object Detection Under Challenging Conditions

Longfei Cui, Hanqing Liu, Mingjie Liu, Endian Lin, Donghong Jiang, Yuhao Wang, Chuang ZhuAAAI 2025

Introduction

Standard RGB cameras are the primary sensors for computer vision, yet they possess a significant "glass jaw": performance degrades rapidly under extreme lighting conditions—such as underexposure in low light or overexposure from glare—and high-speed motion blur. Event cameras offer a compelling alternative, reporting per-pixel brightness changes with microsecond latency and high dynamic range. However, integrating these sensors effectively has been hindered by existing datasets that suffer from low resolution, spatial misalignment, and a lack of diverse "edge case" scenarios.

Methodology

To overcome the challenge of spatial misalignment, the researchers developed a custom Coaxial Imaging System. By utilizing a beam splitter and a coaxial cube, the system ensures that the RGB camera and the Event camera share the exact same optical path, achieving true pixel-level alignment. A hardware signal generator provides a master clock to synchronize both sensors to the microsecond.

Overview of the RGB-Event Coaxial Imaging System, data statistics, and sample visualizations.

Hybrid Annotation Strategy

Labeling objects in pitch-black or blurry RGB frames is nearly impossible for human annotators. To establish accurate ground truth, the team employed an AI reconstruction algorithm (NER-Net) to transform raw event streams into high-clarity grayscale images. Annotators then drew bounding boxes on these reconstructions, which were subsequently mapped back to the pixel-aligned RGB frames.

The PEOD Dataset

The Pixel-aligned Event-RGB Object Detection (PEOD) dataset represents a significant leap in benchmark quality:

  • High Resolution: 1280 × 720 (720p), enabling detection of small, distant objects.
  • Scale: Over 130 sequences containing 72,000 frames and 340,000 manual annotations across six classes (car, bus, truck, two-wheeler, three-wheeler, person).
  • Challenging Conditions: 57.1% of the data is captured in "Illumination Challenge" environments, including night scenes, tunnels, and intense glare.
Comparison of Event and RGB data across various challenging scenarios, demonstrating pixel alignment and complementary information.

Experimental Results and Insights

The authors benchmarked 14 models, revealing critical insights into multi-modal fusion:

  • The "Lazy Fusion" Problem: While fusion models like EOLO achieved the highest overall mAP (26.1), their performance plummeted in extreme conditions. In the toughest lighting, the top event-only model (SMamba, 23.2 mAP) significantly outperformed the top fusion model (EOLO, 11.5 mAP). This suggests that current fusion strategies are "lazy"—they over-rely on RGB textures and fail to ignore RGB "noise" when frames become unreliable.
  • Event-Only Strengths: SMamba, utilizing the Mamba architecture for sequence modeling, proved highly effective for event data, highlighting the importance of temporal modeling in this modality.
  • RGB Vulnerability: The strongest RGB-only baseline, RT-DETR, saw its performance drop from 21.8 mAP overall to just 13.2 mAP in illumination-challenged subsets.

Conclusion and Future Work

PEOD establishes a new standard for high-resolution, pixel-aligned multi-modal detection. The benchmark reveals a massive opportunity for "reliability-aware" fusion research. Future work should focus on adaptive reweighting mechanisms that allow models to dynamically trust event data more heavily when the RGB signal-to-noise ratio drops. Additionally, the continuous nature of PEOD's sequences makes it a prime candidate for advancing multi-object tracking (MOT) in extreme environments.

Summary updated

Abstract

Robust object detection for challenging scenarios increasingly relies on event cameras, yet existing Event-RGB datasets remain constrained by sparse coverage of extreme conditions and low spatial resolution (≤ 640 × 480), which prevents comprehensive evaluation of detectors under challenging scenarios. To address these limitations, we propose PEOD, the first large-scale, pixel-aligned and hign-resolution (1280 × 720) Event-RGB dataset for object detection under challenge conditions. PEOD contains 130+ spatiotemporal-aligned sequences and 340k manual bounding boxes, with 57% of data captured under low-light, overexposure, and high-speed motion. Furthermore, we benchmark 14 methods across three input configurations (Event-based, RGB-based, and Event-RGB fusion) on PEOD. On the full test set and normal subset, fusion-based models achieve the excellent performance. However, in illumination challenge subset, the top event-based model outperforms all fusion models, while fusion models still outperform their RGB-based counterparts, indicating limits of existing fusion methods when the frame modality is severely degraded. PEOD establishes a realistic, high-quality benchmark for multimodal perception and will be publicly released later to facilitate future research.
BibTeX
@misc{cui2025peod,
  author = {Longfei Cui and Hanqing Liu and Mingjie Liu and Endian Lin and Donghong Jiang and Yuhao Wang and Chuang Zhu},
  title = {{PEOD: A Pixel-Aligned Event-RGB Benchmark for Object Detection Under Challenging Conditions}},
  year = {2025},
  eprint = {2511.08140},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.08140},
}
Figure