UnrealEvent: Synthetic Dataset Generation and Annotation for Event-Based Vision

Yentel Schoovaerts, Zijie Ning, Patrick VandewallearXiv 2026 · arXiv

TL;DR

UnrealEvent is a new open-source pipeline built in Unreal Engine 5.5 that automatically generates and annotates synthetic data for event-based vision. By coupling photorealistic rendering with a rule-based event simulator, it allows researchers to create massive, labeled datasets for neuromorphic object detection without the need for manual bounding-box labeling.

What Problem Does This Paper Address?

Event-based cameras (or neuromorphic sensors) are a radical departure from traditional frame-based cameras. Instead of capturing full images at fixed intervals, each pixel operates independently and only reports back when it detects a change in log-luminance. This results in ultra-high temporal resolution, massive dynamic range, and low power consumption—perfect for high-speed robotics or low-light surveillance.

However, progress in event-based object detection is stalled by a "data drought." Real-world event data is expensive to capture and notoriously difficult to annotate manually because the data is sparse and lacks the intuitive visual cues of standard photos. While synthetic data is a logical solution, existing simulators often produce raw event streams without the synchronized bounding-box annotations needed to train supervised models like YOLO.

Methods

The authors introduce a pipeline that integrates directly into Unreal Engine 5.5, leveraging its high-fidelity rendering and the EasySynth plugin.

  1. Event Generation: Rather than using complex (and often data-hungry) learning-based simulators, the authors use a rule-based approach inspired by the V2E model. It calculates per-pixel log-luminance differences ($\Delta L = L_t - L_{t-1}$) between consecutive high-frame-rate (300 fps) renders.
  2. Sensor Realism: To mimic real hardware, the pipeline samples contrast thresholds for each pixel from a normal distribution, emulating the "threshold mismatch" found in physical sensors. It also models shot noise (random events from photon arrival) and leak events (spontaneous noise from sensor circuitry).
  3. Automated Annotation: This is the "secret sauce." The pipeline uses EasySynth’s semantic rendering passes to identify objects in the 3D scene. A custom script then converts these segmentation masks into axis-aligned bounding boxes in the standard YOLO format, perfectly synchronized with the event frames.
Fig. 1: Full simulation pipeline

Datasets

  • UnrealEvent (New): A synthetic dataset comprising 50,692 annotated event frames across 19 sequences. It includes pedestrians and vehicles in diverse scenarios, including challenging night-time variants.
  • eTraM: A real-world event-based traffic monitoring dataset used as the "ground truth" to evaluate how well the synthetic training generalizes to the real world.

Key Insights and Ideas

  • The "Sparsity Imbalance": The authors discovered that event cameras naturally create a class imbalance. Because they only respond to motion/change, slow-moving pedestrians generate far fewer events than fast-moving vehicles. This makes pedestrians inherently harder to detect in event streams—a property that emerged naturally in the synthetic data.
  • Localization vs. Classification: In "zero-shot" tests (applying the synthetic model to real data without any real-world training), the model struggled with confidence but still produced well-aligned bounding boxes. This suggests the pipeline successfully teaches the model where objects are, even if the "texture" of synthetic noise differs from real sensors.
  • Asset Diversity is King: The model performed significantly worse on real pedestrians than on vehicles. The culprit? The synthetic dataset used only a single pedestrian mesh. This highlights that for event-based vision, the variety of 3D shapes and motion patterns is more critical for generalization than photorealistic textures.
Fig. 2: Example pipeline output: a rendered RGB frame (left) and its corresponding event representation (right). Blue and red denote ON (∆L > 0) and OFF (∆L < 0) events.

Key Results

The researchers evaluated a YOLOv8 detector across three regimes:

  • In-Domain (Synthetic): The model was nearly perfect on vehicles (AP50 of 0.991) and strong on pedestrians (0.888).
  • Zero-Shot (Real eTraM): Performance crashed to roughly 0.07 AP50, exposing a massive "sim-to-real" gap common in neuromorphic research.
  • Fine-Tuning: By adding a small subset of real data (2,436 images), performance on real vehicles jumped to 0.606. While this shows the synthetic data provides a solid foundation, it also proves that current simulators still can't fully replace real-world data.

Limitations and Future Work

The authors are transparent about the remaining hurdles:

  • Noise Fidelity: The current noise model is a "proof of concept." Future versions need to include more complex sensor artifacts like hot pixels and the refractory period (the "dead time" a pixel experiences after firing an event).
  • Asset Variety: As noted, the lack of diverse pedestrian models was a major bottleneck for generalization.
  • Temporal Resolution: Currently, the pipeline exports image-based "event frames." Future iterations aim to export raw neuromorphic formats (like .raw or .h5) to preserve the microsecond-level temporal precision that makes event cameras unique.

Summary updated

Abstract

Event-based cameras provide high temporal resolution, low power consumption, and a high dynamic range, but progress in event-based object detection remains constrained by the scarcity of annotated training data. To solve this issue, this paper introduces the tool UnrealEvent, a synthetic dataset-generation pipeline implemented in Unreal Engine 5.5. The pipeline combines a V2E-inspired event-frame generator with EasySynth semantic-rendering passes, allowing rendered level sequences to be converted into paired event representations and YOLO-format bounding-box annotations without manual labeling. Events are generated from per-pixel log-luminance differences between consecutive frames, with contrast thresholds sampled independently from a normal distribution to emulate the fabrication-induced threshold mismatch of real sensors. Using this pipeline, we provide a synthetic dataset and evaluate YOLOv8 detectors under synthetic, zero-shot real, and real fine-tuning training on eTraM. The results show strong in-domain synthetic performance but a substantial synthetic-to-real gap, especially for pedestrians. Fine-tuning on a small real subset improves real-domain performance, while the remaining gap highlights the importance of asset diversity and sensor calibration.
BibTeX
@misc{schoovaerts2026unrealevent,
  author = {Yentel Schoovaerts and Zijie Ning and Patrick Vandewalle},
  title = {{UnrealEvent: Synthetic Dataset Generation and Annotation for Event-Based Vision}},
  year = {2026},
  eprint = {2611.23078},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23078},
}
Figure