Super Resolve Dynamic Scene from Continuous Spike Streams

Jing Zhao, Jiyu Xie, Ruiqin Xiong, Jian Zhang, Zhaofei Yu, Tiejun Huang

Introduction

Conventional digital cameras often struggle with high-speed motion, as they average light over an exposure window, resulting in motion blur. To overcome this, "spike cameras" inspired by the human fovea have been developed. These sensors fire asynchronous binary spikes whenever a pixel accumulates a specific photon threshold, enabling extremely high temporal resolution (tens of thousands of Hz).

However, spike cameras currently suffer from limited spatial resolution. Traditional reconstruction methods typically aim for the sensor's native resolution. Simply applying standard Super-Resolution (SR) techniques to reconstructed low-resolution images often fails because fine details are lost during the initial reconstruction phase. This paper introduces the Motion-Guided Spike Camera Super-resolution (MGSR) framework, the first to perform SR directly from raw spike data.

Methodology

The MGSR framework treats image reconstruction as an optimization problem rather than a simple averaging process. The core innovation is a mathematical formulation that links the light intensity of a scene point to the spikes fired by a sensor pixel.

Visualization of a continuous spike stream, which serves as the raw input for the reconstruction process.

By accounting for relative motion between the scene and the camera, the framework uses motion as a spatial sampling advantage. A single scene point is captured by different pixels over time, providing sub-pixel information. The pipeline involves light inference for initial low-res images, motion estimation via optical flow, and an optimization step using gradient descent to reconstruct the high-resolution intensity image.

Experimental Results

The researchers evaluated MGSR using the PKU-Spike-High-Speed Dataset and new sequences captured with an FSM spike camera (400x250 resolution at 20,000–40,000 Hz). The scenes included both object motion (e.g., a falling doll, high-speed cars and trains) and camera ego-motion.

Qualitatively, MGSR outperforms existing methods like TFP, TFI, and TVS by recovering sharp textures and legible details that others miss. Quantitatively, MGSR consistently achieved the best (lowest) scores in no-reference quality metrics such as BRISQUE and NIQE. For instance, in a 350 km/h "Train" sequence, MGSR scored 3.14 on NIQE, significantly better than TFP (10.62) and TFI (6.49).

Comparison of the proposed MGSR method (f) against various baseline reconstruction techniques (b-e). The proposed approach demonstrates significantly better detail recovery.

Discussion and Conclusion

The key insight of this work is that motion, often viewed as a source of blur, can be leveraged as a "scanning" mechanism for sub-pixel sampling. The paper demonstrates that "direct" SR from spikes is superior to "pipelined" approaches because it preserves the precise timing of photons, which contains high-frequency spatial information.

While highly effective, the method's performance depends on the accuracy of optical flow estimation. Future work may involve integrating deep learning-based motion estimation or end-to-end neural networks to enhance robustness and computational efficiency.

Summary updated

Abstract

Recently, a novel retina-inspired camera, namely spike camera, has shown great potential for recording high-speed dynamic scenes. Unlike conventional digital cameras that compact the visual information within an exposure interval into a single snapshot, the spike camera continuously outputs binary spike streams to record the dynamic scenes, yielding a very high temporal resolution. Most of the existing reconstruction methods for spike camera focus on reconstructing images with the same resolution as spike camera. However, as a trade-off of high temporal resolution, the spatial resolution of spike camera is limited, resulting in inferior details of the reconstruction. To address this issue, we develop a spike camera super-resolution framework, aiming to super resolve high-resolution intensity images from the low-resolution binary spike streams. Due to the relative motion between the camera and the objects to capture, the spikes fired by the same sensor pixel no longer describes the same points in the external scene. In this paper, we exploit the relative motion and derive the relationship between light intensity and each spike, so as to recover the external scene with both high temporal and high spatial resolution. Experimental results demonstrate that the proposed method can reconstruct pleasant high-resolution images from low- resolution spike streams.
BibTeX
@inproceedings{zhao2021super,
  author = {Jing Zhao and Jiyu Xie and Ruiqin Xiong and Jian Zhang and Zhaofei Yu and Tiejun Huang},
  title = {{Super Resolve Dynamic Scene from Continuous Spike Streams}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00253},
}
Figure