High Speed and High Dynamic Range Video with an Event Camera

Henri Rebecq, René Ranftl, Vladlen Koltun, Davide ScaramuzzaIEEE TPAMI 2019 · DOI · OpenReview

Event cameras are a fascinating new type of sensor that capture the world differently than traditional cameras. Instead of taking snapshots at fixed intervals, they report changes in brightness as they happen, leading to incredibly high temporal resolution and dynamic range. However, turning these streams of asynchronous "events" back into usable video has been a significant challenge. This paper introduces a novel deep learning approach that learns to reconstruct high-quality video from event data, opening up new possibilities for applications in robotics, autonomous driving, and more.

TL;DR

This paper presents a recurrent neural network that learns to reconstruct high-quality video from event camera data, overcoming limitations of previous methods by avoiding hand-crafted priors. The approach achieves state-of-the-art results in high-speed and high-dynamic-range video synthesis and demonstrates its utility as an intermediate representation for downstream computer vision tasks.

What Problem Does This Paper Address?

Traditional cameras capture images at fixed frame rates, which can lead to motion blur in fast-moving scenes and struggle with extreme lighting conditions (high dynamic range). Event cameras, on the other hand, offer microsecond-level temporal resolution and a dynamic range of up to 140 dB, making them ideal for challenging scenarios. However, the asynchronous, sparse nature of event data makes reconstructing a dense, temporally consistent video a difficult, ill-posed problem. Existing methods often rely on hand-crafted priors and assumptions about natural images, which can lead to artifacts and limit their performance. This paper aims to bridge this gap by learning a high-quality video reconstruction directly from data, without relying on such priors.

Methods

The core of this work is a novel recurrent convolutional neural network, inspired by the UNet architecture. This network takes event tensors (spatio-temporal grids of events) as input and, along with its internal state, outputs a reconstructed intensity image. Key aspects of their method include:

  • Recurrent Architecture: A stacked ConvLSTM architecture is used to maintain and update an internal state, enabling the network to learn temporal dependencies and produce temporally consistent video. This is crucial for overcoming the challenges of sparse event data.
  • Data-Driven Reconstruction: Unlike prior work, the network is trained end-to-end on simulated event data, learning to reconstruct images without explicit hand-crafted priors.
  • Perceptual Loss: A perceptual loss function (LPIPS) is employed during training. This encourages the reconstructed images to have natural image statistics, leading to visually pleasing results.
  • Temporal Consistency Loss: An explicit temporal consistency loss is introduced to further improve the smoothness and reduce flickering in the reconstructed videos.
  • Event Representation: Events are converted into spatio-temporal voxel grids, with polarity and timestamp information encoded.
  • Post-processing: A simple robust min/max normalization is applied to rescale the output intensities to the full [0, 1] range.

Datasets

  • Training: The network is trained on simulated event data generated using the ESIM simulator. MS-COCO images are used as the basis for these simulations, allowing for a wide variety of scenes. Data augmentation techniques, including random contrast thresholds, are used to improve generalization.
  • Evaluation: The primary evaluation is performed on real event data from the Event Camera Dataset [38]. Additional qualitative evaluations are conducted on datasets from [2], [5], [43], and [45]. For downstream applications, datasets like N-MNIST, N-CARS, and N-Caltech101 are used, which are event-based versions of standard image datasets.

Key Insights and Ideas

  • Learning Priors is Powerful: The paper demonstrates that learning reconstruction priors directly from data, rather than relying on hand-crafted ones, leads to significantly better results and fewer artifacts.
  • Recurrent Networks Excel at Temporal Consistency: The use of ConvLSTMs is shown to be effective in maintaining temporal coherence in the reconstructed video, a critical challenge with sparse event data.
  • Synthetic Data Generalizes Well: The network trained on simulated data generalizes surprisingly well to real event data, a crucial finding for enabling practical applications.
  • Event Data as a Bridge: Reconstructed images from event data can serve as a powerful intermediate representation, allowing standard computer vision algorithms to be applied directly to event camera inputs. This opens up a vast ecosystem of existing tools and techniques.
  • High-Speed and HDR are Achievable: The method successfully reconstructs videos of extremely fast phenomena (thousands of frames per second) and scenes with very high dynamic range, showcasing the unique capabilities of event cameras.

Key Results

  • Image Quality: The proposed method outperforms state-of-the-art methods by a large margin, achieving an average of 24% increase in SSIM and a 22% decrease in LPIPS on real event data.
  • High-Speed Video: The network can synthesize videos at over 5,000 frames per second, revealing details invisible to the naked eye or conventional cameras.
  • High Dynamic Range: Reconstructions preserve the high dynamic range of event cameras, successfully capturing details in scenes with extreme lighting variations.
  • Color Video: A simple extension allows for color video reconstruction without retraining the network.
  • Downstream Applications: When used as an intermediate representation, the reconstructed images enable state-of-the-art performance in object classification and visual-inertial odometry, even outperforming methods specifically designed for event data.
  • Computational Efficiency: While not as fast as some event-integration methods, the approach can run in real-time and offers a good trade-off between quality and performance.

Limitations and Future Work

  • Computational Cost: The reconstruction process is not as fast as some event-integration methods, though it is real-time capable. The authors suggest that hardware optimization and exploiting event tensor sparsity could improve efficiency.
  • Generalization to Diverse Sensors: While the network trained on simulated data generalizes well, performance can degrade when applied to event data from sensors with significantly different statistics or illumination conditions. Future work could explore more robust training strategies or sensor-specific fine-tuning.
  • Memory Decay: In the absence of events, the network gradually decays image intensity, a behavior that is not hardcoded but learned. The effective memory length is tied to the training data's optical flow distribution. Future work could tailor training data to specific applications (e.g., autonomous driving) to improve memory retention in low-event-rate regions.
  • Further Downstream Tasks: The authors suggest exploring other downstream applications beyond object classification and VIO, such as object detection and monocular depth prediction, which have not been demonstrated with event data before.

Summary updated

Abstract

Event cameras are novel sensors that report brightness changes in the form of a stream of asynchronous "events" instead of intensity frames. They offer significant advantages with respect to conventional cameras: high temporal resolution, high dynamic range, and no motion blur. While the stream of events encodes in principle the complete visual signal, the reconstruction of an intensity image from a stream of events is an ill-posed problem in practice. Existing reconstruction approaches are based on hand-crafted priors and strong assumptions about the imaging process as well as the statistics of natural images. In this work we propose to learn to reconstruct intensity images from event streams directly from data instead of relying on any hand-crafted priors. We propose a novel recurrent network to reconstruct videos from a stream of events, and train it on a large amount of simulated event data. During training we propose to use a perceptual loss to encourage reconstructions to follow natural image statistics. We further extend our approach to synthesize color images from color event streams. Our network surpasses state-of-the-art reconstruction methods by a large margin in terms of image quality (> 20%), while comfortably running in real-time. We show that the network is able to synthesize high framerate videos (> 5,000 frames per second) of high-speed phenomena (e.g. a bullet hitting an object) and is able to provide high dynamic range reconstructions in challenging lighting conditions. We also demonstrate the effectiveness of our reconstructions as an intermediate representation for event data. We show that off-the-shelf computer vision algorithms can be applied to our reconstructions for tasks such as object classification and visual-inertial odometry and that this strategy consistently outperforms algorithms that were specifically designed for event data.
BibTeX
@article{rebecq2019high,
  author = {Henri Rebecq and René Ranftl and Vladlen Koltun and Davide Scaramuzza},
  title = {{High Speed and High Dynamic Range Video with an Event Camera}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2019},
  doi = {10.1109/tpami.2019.2963386},
}
Figure