TL;DR
UnrealEvent is a new open-source pipeline built in Unreal Engine 5.5 that automatically generates and annotates synthetic data for event-based vision. By coupling photorealistic rendering with a rule-based event simulator, it allows researchers to create massive, labeled datasets for neuromorphic object detection without the need for manual bounding-box labeling.
What Problem Does This Paper Address?
Event-based cameras (or neuromorphic sensors) are a radical departure from traditional frame-based cameras. Instead of capturing full images at fixed intervals, each pixel operates independently and only reports back when it detects a change in log-luminance. This results in ultra-high temporal resolution, massive dynamic range, and low power consumption—perfect for high-speed robotics or low-light surveillance.
However, progress in event-based object detection is stalled by a "data drought." Real-world event data is expensive to capture and notoriously difficult to annotate manually because the data is sparse and lacks the intuitive visual cues of standard photos. While synthetic data is a logical solution, existing simulators often produce raw event streams without the synchronized bounding-box annotations needed to train supervised models like YOLO.
Methods
The authors introduce a pipeline that integrates directly into Unreal Engine 5.5, leveraging its high-fidelity rendering and the EasySynth plugin.
- Event Generation: Rather than using complex (and often data-hungry) learning-based simulators, the authors use a rule-based approach inspired by the V2E model. It calculates per-pixel log-luminance differences ($\Delta L = L_t - L_{t-1}$) between consecutive high-frame-rate (300 fps) renders.
- Sensor Realism: To mimic real hardware, the pipeline samples contrast thresholds for each pixel from a normal distribution, emulating the "threshold mismatch" found in physical sensors. It also models shot noise (random events from photon arrival) and leak events (spontaneous noise from sensor circuitry).
- Automated Annotation: This is the "secret sauce." The pipeline uses EasySynth’s semantic rendering passes to identify objects in the 3D scene. A custom script then converts these segmentation masks into axis-aligned bounding boxes in the standard YOLO format, perfectly synchronized with the event frames.
Datasets
- UnrealEvent (New): A synthetic dataset comprising 50,692 annotated event frames across 19 sequences. It includes pedestrians and vehicles in diverse scenarios, including challenging night-time variants.
- eTraM: A real-world event-based traffic monitoring dataset used as the "ground truth" to evaluate how well the synthetic training generalizes to the real world.
Key Insights and Ideas
- The "Sparsity Imbalance": The authors discovered that event cameras naturally create a class imbalance. Because they only respond to motion/change, slow-moving pedestrians generate far fewer events than fast-moving vehicles. This makes pedestrians inherently harder to detect in event streams—a property that emerged naturally in the synthetic data.
- Localization vs. Classification: In "zero-shot" tests (applying the synthetic model to real data without any real-world training), the model struggled with confidence but still produced well-aligned bounding boxes. This suggests the pipeline successfully teaches the model where objects are, even if the "texture" of synthetic noise differs from real sensors.
- Asset Diversity is King: The model performed significantly worse on real pedestrians than on vehicles. The culprit? The synthetic dataset used only a single pedestrian mesh. This highlights that for event-based vision, the variety of 3D shapes and motion patterns is more critical for generalization than photorealistic textures.
Key Results
The researchers evaluated a YOLOv8 detector across three regimes:
- In-Domain (Synthetic): The model was nearly perfect on vehicles (AP50 of 0.991) and strong on pedestrians (0.888).
- Zero-Shot (Real eTraM): Performance crashed to roughly 0.07 AP50, exposing a massive "sim-to-real" gap common in neuromorphic research.
- Fine-Tuning: By adding a small subset of real data (2,436 images), performance on real vehicles jumped to 0.606. While this shows the synthetic data provides a solid foundation, it also proves that current simulators still can't fully replace real-world data.
Limitations and Future Work
The authors are transparent about the remaining hurdles:
- Noise Fidelity: The current noise model is a "proof of concept." Future versions need to include more complex sensor artifacts like hot pixels and the refractory period (the "dead time" a pixel experiences after firing an event).
- Asset Variety: As noted, the lack of diverse pedestrian models was a major bottleneck for generalization.
- Temporal Resolution: Currently, the pipeline exports image-based "event frames." Future iterations aim to export raw neuromorphic formats (like .raw or .h5) to preserve the microsecond-level temporal precision that makes event cameras unique.
Summary updated