Learn to See by Events: RGB Frame Synthesis from Event Cameras

Stefano Pini, Guido Borghi, Roberto Vezzani, Rita CucchiaraarXiv 2018 · arXiv

Abstract

Event cameras, or neuromorphic sensors, provide high temporal resolution and dynamic range by capturing pixel-wise brightness changes asynchronously. However, their lack of static texture and color information limits their compatibility with standard computer vision algorithms. This paper introduces a deep learning framework that bridges this gap by synthesizing high-quality RGB frames from event streams and periodic "key-frames." Using a conditional GAN and a recurrent module, the system generates realistic video sequences capable of supporting downstream tasks like object detection and semantic segmentation.

Introduction: The Neuromorphic Advantage and the Reconstruction Gap

Traditional cameras capture full frames at fixed intervals, often missing high-speed motion or failing in high-contrast environments. Event cameras address these limitations by reporting only changes in brightness as they occur. While advantageous for low-power and high-speed applications, the resulting data is sparse and lacks the color and texture required for mature vision models.

An example of an event frame, capturing sparse pixel-wise brightness changes rather than full-scene textures.

Existing reconstruction methods often rely on hand-crafted filters or complex optimization loops, which frequently result in blurry or grayscale outputs. This work aims to provide a robust method for full-color RGB synthesis by leveraging the strengths of both traditional and neuromorphic sensors.

Methodology

The proposed architecture consists of two primary components designed to handle spatial detail and temporal consistency.

1. Generative Module (G)

The core of the system is a conditional Generative Adversarial Network (cGAN) with a U-Net backbone. It takes an initial RGB "key-frame" and a 2D "event frame" (representing events over a short window) to predict the subsequent RGB frame. Adversarial training ensures the synthesis of sharp textures rather than blurry averages.

2. Recurrent Module (R)

To prevent flickering and long-term drift, a Convolutional LSTM (ConvLSTM) layer is integrated. This module processes sequences of generated frames to refine details and maintain temporal coherence across the video.

The proposed framework synthesizes a sequence of RGB frames by combining an initial key-frame with a stream of event frames over time.

Simulated Event Training

A key innovation is the use of simulated event data. By calculating the logarithmic difference between consecutive frames in standard RGB datasets (like Kitti and Cityscapes), the authors generated large-scale training data, overcoming the scarcity of annotated real-world event datasets. This approach allows the model to learn complex color and texture synthesis from high-fidelity sources.

Experimental Evaluation

The framework was tested on both real-world (DDD17, MVSEC) and simulated (Kitti, Cityscapes) datasets.

Quantitative Performance

On the DDD17 dataset, the method achieved a PSNR of 29.428 and an SSIM of 0.884, significantly outperforming previous state-of-the-art methods. Perceptual quality, measured via LPIPS, also showed marked improvement, indicating that the synthesized images are visually convincing to human observers.

Downstream Task Compatibility

The synthesized frames were high-quality enough to be used directly with off-the-shelf vision models. For instance, object detection using YOLOv3 and segmentation using DeepLab on synthesized Cityscapes frames nearly matched performance on original ground-truth frames, achieving an mIoU of 0.926.

Discussion and Future Work

The study demonstrates that models trained on simulated events generalize surprisingly well to real-world sensors. However, the system currently relies on periodic key-frames to provide initial color and context; if the interval between these frames is too long, error accumulation (drift) can occur. While the current implementation runs at approximately 20Hz on an NVIDIA 1080Ti, future optimizations are needed to reach the microsecond-level latency potential of event cameras on edge devices.

Conclusion

By combining the strengths of traditional RGB sensors and neuromorphic event cameras, this framework enables high-fidelity vision in challenging conditions while maintaining compatibility with existing computer vision pipelines. This research marks a significant step toward making neuromorphic sensors practical for mainstream autonomous systems.

Summary updated

Abstract

Event cameras are biologically-inspired sensors that gather the temporal evolution of the scene. They capture pixel-wise brightness variations and output a corresponding stream of asynchronous events. Despite having multiple advantages with respect to traditional cameras, their use is partially prevented by the limited applicability of traditional data processing and vision algorithms. To this aim, we present a framework which exploits the output stream of event cameras to synthesize RGB frames, relying on an initial or a periodic set of color key-frames and the sequence of intermediate events. Differently from existing work, we propose a deep learning-based frame synthesis method, consisting of an adversarial architecture combined with a recurrent module. Qualitative results and quantitative per-pixel, perceptual, and semantic evaluation on four public datasets confirm the quality of the synthesized images.
BibTeX
@misc{pini2018learn,
  author = {Stefano Pini and Guido Borghi and Roberto Vezzani and Rita Cucchiara},
  title = {{Learn to See by Events: RGB Frame Synthesis from Event Cameras}},
  year = {2018},
  eprint = {1812.02041},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.02041},
}
Figure