High Frame Rate Video Reconstruction Based on an Event Camera

Liyuan Pan, Richard Hartley, Cedric Scheerlinck, Miaomiao Liu, Xin Yu, Yuchao DaiIEEE TPAMI 2022 · OpenReview

TL;DR

This paper introduces a novel method, mEDI, to reconstruct high-quality, high frame rate sharp videos from low-frame-rate, motion-blurred images and event data captured by DAVIS cameras. By modeling the blur formation process and leveraging the precise temporal information from events, mEDI effectively removes blur and enhances temporal resolution, outperforming existing state-of-the-art techniques.

What Problem Does This Paper Address?

Traditional cameras capture images at a relatively low frame rate, leading to motion blur in dynamic scenes. Event cameras, like the DAVIS, offer microsecond-level temporal resolution by detecting intensity changes, but their output intensity frames still suffer from blur. Existing methods for reconstructing high frame rate videos from event data often ignore the blur present in the intensity frames or fail to effectively combine event and image information for deblurring. This paper addresses the challenge of reconstructing sharp, high frame rate videos by simultaneously deblurring and enhancing temporal resolution, specifically by leveraging the complementary information from both blurred intensity frames and event data.

Methods

The core contribution is the Event-based Double Integral (EDI) model, and its extension to the multiple Event-based Double Integral (mEDI) model.

  • EDI Model: This model establishes a relationship between a blurred image, the latent sharp image, and the integrated event data. It models the blurred image formation as an integral of latent sharp images over the exposure time, where events provide precise temporal information about intensity changes. The model is solved by optimizing a non-convex energy function.
  • mEDI Model: To overcome the limitations of using a single blurred frame (like noise accumulation and flickering), the mEDI model extends EDI to handle multiple blurred images and their corresponding events. This allows for more robust reconstruction and smoother video generation by considering a sequence of frames.
  • Optimization: The paper proposes an efficient optimization approach to solve the mEDI model. This involves formulating the problem as a system of linear equations and solving it using LU decomposition and back-substitution, significantly reducing computational complexity. The optimization primarily focuses on finding an optimal contrast threshold parameter 'c' that balances deblurring and reconstruction quality.

Datasets

The paper evaluates their method on:

  • Synthetic Dataset: Based on the GoPro blur dataset [53], which provides ground truth videos to generate blurred images and event data.
  • Real Dataset:
    • A public Event-Camera dataset [58].
  • Their own real blur event dataset captured using a DAVIS240 camera under various conditions (indoor, outdoor, low lighting, dynamic motion).
  • The Color Event Camera Dataset (CED) [62].

Key Insights and Ideas

  • Blur as an Integral of Latent Images: The paper frames motion blur not just as an artifact, but as an integral of the underlying sharp frames over time. This perspective allows for modeling the blur generation process.
  • Events as Precise Temporal Markers: Events, with their microsecond accuracy, provide crucial temporal information that complements the low-resolution, blurred intensity frames. They act as precise indicators of when and where intensity changes occur.
  • Joint Deblurring and Reconstruction: Instead of deblurring first and then reconstructing, the mEDI model jointly addresses both challenges, leading to superior results by fully exploiting the synergy between image and event data.
  • Efficient Optimization for Multi-frame Reconstruction: The LU decomposition-based solution for the mEDI model is a key innovation, enabling efficient processing of multiple frames and reducing computational burden.
  • Robustness to Lighting and Motion: The mEDI model demonstrates robustness in challenging conditions like low lighting and high dynamic motion, where traditional methods often struggle.

Key Results

  • Quantitative Improvement: The mEDI model achieves state-of-the-art performance, outperforming existing methods in terms of PSNR and SSIM on both synthetic and real datasets. For instance, on the synthetic dataset, their mEDI model achieves a PSNR of 28.83 dB and SSIM of 0.9098, surpassing other methods.
  • High Frame Rate Reconstruction: The reconstructed videos achieve significantly higher frame rates (up to 200 times greater than the input) while maintaining sharpness and temporal consistency.
  • Qualitative Superiority: Visual results show that mEDI produces sharper images with richer textures and better consistency compared to event-only methods or methods that don't jointly handle blur and temporal reconstruction. Figures 1, 5, 6, 9, and 10 visually demonstrate these improvements.
  • Robustness: The method performs well in low lighting and complex dynamic scenes, as shown in Figure 9.

Limitations and Future Work

  • Extreme Lighting Changes: The paper acknowledges that extreme lighting changes (e.g., sudden on/off of lights, moving between very dark and very bright scenes) can still degrade performance due to the limited dynamic range of the intensity image.
  • Event Error Accumulation: While mEDI mitigates some issues, accumulated noise from sparse or erroneous event data can still reduce the quality of reconstructed images, especially in areas with less motion.
  • Future Work: The authors suggest exploring methods to handle extreme lighting conditions and further improving robustness to event errors. They also highlight the potential for extending their work to color event cameras.

Summary updated

Abstract

Event-based cameras measure intensity changes (called ‘ <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">events</i> ’) with microsecond accuracy under high-speed motion and challenging lighting conditions. With the ‘active pixel sensor’ (APS), the ‘Dynamic and Active-pixel Vision Sensor’ (DAVIS) allows the simultaneous output of intensity frames and events. However, the output images are captured at a relatively low frame rate and often suffer from motion blur. A blurred image can be regarded as the integral of a sequence of latent images, while events indicate changes between the latent images. Thus, we are able to model the blur-generation process by associating event data to a latent sharp image. Based on the abundant event data alongside a low frame rate, easily blurred images, we propose a simple yet effective approach to reconstruct high-quality and high frame rate sharp videos. Starting with a single blurred frame and its event data from DAVIS, we propose the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Event-based Double Integral (EDI)</i> model and solve it by adding regularization terms. Then, we extend it to <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">multiple Event-based Double Integral (mEDI)</i> model to get more smooth results based on multiple images and their events. Furthermore, we provide a new and more efficient solver to minimize the proposed energy model. By optimizing the energy function, we achieve significant improvements in removing blur and the reconstruction of a high temporal resolution video. The video generation is based on solving a simple non-convex optimization problem in a single scalar variable. Experimental results on both synthetic and real datasets demonstrate the superiority of our <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">mEDI</i> model and optimization method compared to the state-of-the-art.
BibTeX
@article{pan2022high,
  author = {Liyuan Pan and Richard Hartley and Cedric Scheerlinck and Miaomiao Liu and Xin Yu and Yuchao Dai},
  title = {{High Frame Rate Video Reconstruction Based on an Event Camera}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=uapqymC6dq_},
}
Figure