ControlEvents: Controllable Synthesis of Event Camera Datawith Foundational Prior from Image Diffusion Models

Yixuan Hu, Yuxuan Xue, Simon Klenk, Daniel Cremers, Gerard Pons-Moll

Abstract

Event cameras capture pixel-level brightness changes with high temporal resolution and dynamic range, yet their adoption is hindered by the difficulty of obtaining large-scale labeled datasets. We introduce ControlEvents, a generative framework that repurposes pre-trained image diffusion models (e.g., Stable Diffusion) for event data synthesis. By conditioning generation on text, 2D skeletons, or 3D body poses, ControlEvents produces high-fidelity synthetic event streams that capture real-world sensor noise and idiosyncrasies. Our approach significantly improves performance on downstream tasks like object recognition and human pose estimation, demonstrating the power of foundational image priors in the event-based vision domain.

Introduction: The Data Hunger of Event Cameras

Unlike traditional frame-based cameras, event cameras only record asynchronous changes in brightness. This unique sensing modality provides advantages in high-speed and high-dynamic-range scenarios. However, training deep learning models for event-based vision requires massive amounts of labeled data. Manual labeling is prohibitively expensive and difficult because raw event streams are not easily interpretable by humans. While simulators offer a potential solution, they often fail to model the complex noise and sensor characteristics of real hardware, leading to a significant "domain gap" that limits the transferability of models trained on synthetic data.

Methodology: Leveraging Foundational Priors

ControlEvents addresses the data scarcity problem by treating event generation as a controllable image synthesis task. The framework converts asynchronous event streams into 2D histograms—3-channel images where red and blue channels represent positive and negative events, respectively.

Figure 1: Overview of the ControlEvents framework, showing diverse control conditions (text, skeletons, SMPL) and the resulting high-quality event generation across various categories.

Instead of training a generative model from scratch, ControlEvents leverages the extensive visual knowledge embedded in large-scale image models like Stable Diffusion and ControlNet. The system supports multiple control modalities:

  • Text-to-Event: Fine-tuning the diffusion model to generate events based on descriptive prompts (e.g., "a photo of a minibus").
  • Skeleton-to-Event: Utilizing 2D human skeletons as spatial maps to guide the generation of motion-centric events.
  • 3D Pose-to-Event: Converting 3D SMPL model parameters into 2D normal maps, providing the diffusion model with surface geometry to synthesize realistic human motion events.

By fine-tuning on a small set of real event data, the model learns to map the structural priors of the foundational model to the specific visual style and noise distributions of event sensors.

Results and Key Insights

The researchers evaluated ControlEvents across benchmarks for object recognition (N-ImageNet), 2D skeleton estimation (DHP-19), and 3D body pose estimation (Event-HPE).

Performance Gains

The inclusion of synthetic data from ControlEvents consistently improved model accuracy. On N-ImageNet, object recognition accuracy rose from 39.33% to 45.33%. In 3D human pose estimation, the Mean Per Joint Position Error (3D-MPJPE) dropped from 27.30mm to 18.65mm, outperforming traditional simulators like SMPL-ESIM.

The Power of the Prior

A key finding is the model's ability to generate events for objects never seen in event format during training. Because the underlying Stable Diffusion model contains priors for billions of RGB images, it can synthesize events for a "ferry" or "windsor chair" simply from a text prompt. This zero-shot capability is a significant advancement for the field.

Figure 6: High-fidelity event generation for a human pose using ControlEvents, capturing realistic structure and sensor-like noise.

Noise as a Feature

Unlike traditional simulators that produce "too clean" data, ControlEvents learns the distribution of real sensor noise. Quantitatively, the Frechet Inception Distance (FID) for ControlEvents was 15.55, compared to 196.41 for Event-GAN, indicating a much closer alignment with real-world event data distributions.

Conclusion

ControlEvents represents a paradigm shift from physics-based simulation to learning-based distribution modeling for event data. By bridging the gap between frame-based foundational models and event-based sensors, it provides a scalable way to generate diverse, labeled datasets. Future work will explore the use of video diffusion models to better capture the continuous temporal nature of event streams.

Summary updated

BibTeX
@inproceedings{hu2026controlevents,
  author = {Yixuan Hu and Yuxuan Xue and Simon Klenk and Daniel Cremers and Gerard Pons-Moll},
  title = {{ControlEvents: Controllable Synthesis of Event Camera Datawith Foundational Prior from Image Diffusion Models}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2026},
}
Figure