Abstract
Event cameras, or neuromorphic sensors, provide high temporal resolution and dynamic range by capturing pixel-wise brightness changes asynchronously. However, their lack of static texture and color information limits their compatibility with standard computer vision algorithms. This paper introduces a deep learning framework that bridges this gap by synthesizing high-quality RGB frames from event streams and periodic "key-frames." Using a conditional GAN and a recurrent module, the system generates realistic video sequences capable of supporting downstream tasks like object detection and semantic segmentation.
Introduction: The Neuromorphic Advantage and the Reconstruction Gap
Traditional cameras capture full frames at fixed intervals, often missing high-speed motion or failing in high-contrast environments. Event cameras address these limitations by reporting only changes in brightness as they occur. While advantageous for low-power and high-speed applications, the resulting data is sparse and lacks the color and texture required for mature vision models.
Existing reconstruction methods often rely on hand-crafted filters or complex optimization loops, which frequently result in blurry or grayscale outputs. This work aims to provide a robust method for full-color RGB synthesis by leveraging the strengths of both traditional and neuromorphic sensors.
Methodology
The proposed architecture consists of two primary components designed to handle spatial detail and temporal consistency.
1. Generative Module (G)
The core of the system is a conditional Generative Adversarial Network (cGAN) with a U-Net backbone. It takes an initial RGB "key-frame" and a 2D "event frame" (representing events over a short window) to predict the subsequent RGB frame. Adversarial training ensures the synthesis of sharp textures rather than blurry averages.
2. Recurrent Module (R)
To prevent flickering and long-term drift, a Convolutional LSTM (ConvLSTM) layer is integrated. This module processes sequences of generated frames to refine details and maintain temporal coherence across the video.
Simulated Event Training
A key innovation is the use of simulated event data. By calculating the logarithmic difference between consecutive frames in standard RGB datasets (like Kitti and Cityscapes), the authors generated large-scale training data, overcoming the scarcity of annotated real-world event datasets. This approach allows the model to learn complex color and texture synthesis from high-fidelity sources.
Experimental Evaluation
The framework was tested on both real-world (DDD17, MVSEC) and simulated (Kitti, Cityscapes) datasets.
Quantitative Performance
On the DDD17 dataset, the method achieved a PSNR of 29.428 and an SSIM of 0.884, significantly outperforming previous state-of-the-art methods. Perceptual quality, measured via LPIPS, also showed marked improvement, indicating that the synthesized images are visually convincing to human observers.
Downstream Task Compatibility
The synthesized frames were high-quality enough to be used directly with off-the-shelf vision models. For instance, object detection using YOLOv3 and segmentation using DeepLab on synthesized Cityscapes frames nearly matched performance on original ground-truth frames, achieving an mIoU of 0.926.
Discussion and Future Work
The study demonstrates that models trained on simulated events generalize surprisingly well to real-world sensors. However, the system currently relies on periodic key-frames to provide initial color and context; if the interval between these frames is too long, error accumulation (drift) can occur. While the current implementation runs at approximately 20Hz on an NVIDIA 1080Ti, future optimizations are needed to reach the microsecond-level latency potential of event cameras on edge devices.
Conclusion
By combining the strengths of traditional RGB sensors and neuromorphic event cameras, this framework enables high-fidelity vision in challenging conditions while maintaining compatibility with existing computer vision pipelines. This research marks a significant step toward making neuromorphic sensors practical for mainstream autonomous systems.
Summary updated