Abstract
Recently, event-based vision sensors have gained attention for autonomous driving applications, as conventional RGB cameras face limitations in handling challenging dynamic conditions. However, the availability of real-world and synthetic event-based vision datasets remains limited. In response to this gap, we present SEVD, a first-of-its-kind multi-view ego, and fixed perception synthetic event-based dataset using multiple dynamic vision sensors within the CARLA simulator. Data sequences are recorded across diverse lighting (noon, nighttime, twilight) and weather conditions (clear, cloudy, wet, rainy, foggy) with domain shifts (discrete and continuous). SEVD spans urban, suburban, rural, and highway scenes featuring various classes of objects (car, truck, van, bicycle, motorcycle, and pedestrian). Alongside event data, SEVD includes RGB imagery, depth maps, optical flow, semantic, and instance segmentation, facilitating a comprehensive understanding of the scene. Furthermore, we evaluate the dataset using state-of-the-art event-based (RED, RVT) and frame-based (YOLOv8) methods for traffic participant detection tasks and provide baseline benchmarks for assessment. Additionally, we conduct experiments to assess the synthetic event-based dataset's generalization capabilities.
Introduction
Standard RGB cameras are the workhorses of computer vision but struggle with high-speed motion blur and extreme lighting conditions. Event cameras, or Dynamic Vision Sensors (DVS), address these limitations by mimicking biological retinas—reporting pixel-level intensity changes asynchronously. This results in microsecond resolution and high dynamic range. Despite their promise, the development of event-based algorithms is hindered by a lack of large-scale, annotated datasets. Existing datasets are often limited to single-view ego-perception. SEVD bridges this gap by providing a comprehensive multi-view dataset that includes both vehicle-mounted (ego) and infrastructure-mounted (fixed) perspectives, essential for advancing Vehicle-to-Infrastructure (V2I) communication.
Methods: The SEVD Framework
The Synthetic Event-based Vision Dataset (SEVD) was generated using the CARLA simulator to create a high-fidelity playground for neuromorphic perception. Unlike many datasets that provide limited modalities, SEVD features a "full-stack" sensor suite.
Sensor Configuration and Modalities
The dataset utilizes two primary perception modes:
- Ego Perception: A vehicle equipped with six DVS cameras providing a full 360° field of view.
- Fixed Perception: Four DVS cameras strategically placed at intersections, roundabouts, and underpasses to simulate smart infrastructure.
For every frame, SEVD provides raw event streams, RGB imagery, depth maps, optical flow, and both semantic and instance segmentation. Navigation data from GNSS and IMU are also included for precise tracking. The simulation covers diverse environments (urban, rural, highway) under three lighting scenarios (noon, night, twilight) and five weather conditions (clear, cloudy, wet, rainy, foggy).
Dataset Characteristics
SEVD comprises over 58 total hours of data (31h ego, 27h fixed) with 9 million 2D and 3D bounding box annotations. It covers six object classes: car, truck, van, bicycle, motorcycle, and pedestrian. Compared to existing benchmarks like GEN1 or 1Mpx, SEVD offers significantly higher data volume, more diverse viewpoints, and a richer set of ground-truth modalities that are difficult to obtain in real-world settings.
Experimental Results and Benchmarks
Baselines were established using state-of-the-art event detectors (RED and RVT) and a frame-based detector (YOLOv8).
- Perception Performance: The Recurrent Vision Transformer (RVT) consistently outperformed the REcurrent Detection (RED) model. In noon-time fixed perception, RVT achieved a Mean Average Precision (mAP@50) of 0.683, which dropped to 0.496 at night. In ego-perception urban settings, RVT reached an mAP of 0.515.
- Synthetic-to-Real Generalization: A key finding was the validation of synthetic data fidelity. An RVT model trained on SEVD was tested on real-world intersection data from a Prophesee EVK4 HD event camera, achieving an mAP of 0.391. This demonstrates that the features learned from synthetic events are physically meaningful and transferable to real-world hardware.
- Challenges: Pedestrian detection remains a significant challenge due to their slower motion and smaller visual profiles compared to vehicles. While event sensors maintain utility at night (outperforming frame-based methods in high-contrast scenarios), a performance gap between day and night perception persists.
Conclusion and Future Work
SEVD provides a massive, high-fidelity resource for training robust neuromorphic perception algorithms. By combining ego and fixed perspectives with diverse environmental conditions, it facilitates the development of next-generation autonomous driving and smart infrastructure systems. Future research will focus on multi-view data fusion and addressing class imbalances to improve detection for "long-tail" edge cases like bicycles and motorcycles.
Dataset and code: https://eventbasedvision.github.io/SEVD
Summary updated