NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras

Lin Zhu, Jianing Li, Xiao Wang, Tiejun Huang, Yonghong Tian

Abstract

NeuSpike-Net is a deep learning framework that mimics the human visual system by fusing data from two distinct neuromorphic sensors: the Dynamic Vision Sensor (DVS) and the Spike Camera. By combining the high-speed motion sensitivity of DVS with the fine-grained texture sampling of Spike Cameras, the proposed architecture achieves high-quality, high-dynamic-range (HDR) video reconstruction that surpasses the capabilities of either sensor in isolation. This paper details the dual-path encoder design and the novel Feature Fusion Module (FFM) that enables robust performance in extreme conditions such as ultra-high-speed motion and low-light environments.

Introduction

Capturing high-speed scenes with high dynamic range is a major challenge for conventional frame-based cameras, which suffer from motion blur and over/under-exposure. Bio-inspired neuromorphic sensors offer a solution, but the field has historically been split between two distinct sensor types:

  1. Dynamic Vision Sensors (DVS): Similar to the periphery of the human retina, these detect temporal changes in light. They are excellent for motion and HDR but struggle to reconstruct static textures.
  2. Spike Cameras: Similar to the fovea, these accumulate light intensity to fire spikes. They capture rich texture but are highly sensitive to noise and have a limited dynamic range.

NeuSpike-Net addresses the gap of how to effectively fuse these two heterogeneous, asynchronous data streams to create a "super-sensor" capable of reconstructing high-fidelity video.

Bio-inspired pipeline of NeuSpike-Net, mimicking the human visual system's dual-path processing.

Methods

The authors propose NeuSpike-Net, a U-Net-style architecture featuring a dual-path encoder and a shared decoder.

Adaptive Data Representation

To handle the asynchronous nature of the data, Spike data is represented using a temporal Inter-Spike Interval (ISI) map, while Event data is processed through an "event integration" model inspired by the integrate-and-fire neuron.

Dual-Path Encoding and Fusion

The architecture utilizes two specialized paths:

  • Motion Path: Processes DVS events to capture high-speed movement and HDR information.
  • Texture Path: Processes Spike data to extract fine spatial details.

The core innovation is the Feature Fusion Module (FFM). Instead of simple concatenation, the FFM uses Channel and Spatial Attention to learn dynamic weights for features from both paths. For instance, the network learns to rely more on DVS in dark regions where Spike noise is high, or on Spike data when the scene is static and DVS is "blind."

Detailed architecture of NeuSpike-Net featuring dual-path encoders and Feature Fusion Modules.

Experimental Evaluation

The framework was evaluated using both simulated and real-world datasets. The simulated dataset was derived from KITTI, upsampled to 22,500 FPS to provide ground truth. The real-world dataset was captured using a custom hybrid rig aligning a Vidar spike camera and a DAVIS 346 event camera.

Key Results

  • Quantitative Superiority: On simulated "Complex" scenes (high speed + low light), NeuSpike-Net achieved a PSNR of 27.40 and SSIM of 0.8510, significantly outperforming DVS-only methods like E2VID (PSNR 12.85) and Spike-only methods like SNM (PSNR 26.44).
  • Real-World Performance: In tests with a fan spinning at 2600 RPM, NeuSpike-Net was the only method able to reconstruct clear, legible letters on the fan blades without significant artifacts.
  • HDR Handling: The system successfully reconstructed details in scenes where traditional sensors would have seen only black shadows or blown-out highlights.

Conclusion

NeuSpike-Net demonstrates that the DVS and Spike camera are naturally complementary, much like the human eye's peripheral and foveal vision. While the system achieves state-of-the-art results, future work will address challenges in sub-pixel hardware alignment and computational complexity to enable deployment on edge devices like drones or autonomous vehicles.

Summary updated

Abstract

Neuromorphic vision sensor is a new bio-inspired imaging paradigm that emerged in recent years, which continuously sensing luminance intensity and firing asynchronous spikes (events) with high temporal resolution. Typically, there are two types of neuromorphic vision sensors, namely dynamic vision sensor (DVS) and spike camera. From the perspective of bio-inspired sampling, DVS only perceives movement by imitating the retinal periphery, while the spike camera was developed to perceive fine textures by simulating the fovea. It is meaningful to explore how to combine two types of neuromorphic cameras to reconstruct high quality image like human vision. In this paper, we propose a NeuSpike-Net to learn both the high dynamic range and high motion sensitivity of DVS and the full texture sampling of spike camera to achieve high-speed and high dynamic image reconstruction. We propose a novel representation to effectively extract the temporal information of spike and event data. By introducing the feature fusion module, the two types of neuromorphic data achieve complementary to each other. The experimental results on the simulated and real datasets demonstrate that the proposed approach is effective to reconstruct high-speed and high dynamic range images via the combination of spike and event data.
BibTeX
@inproceedings{zhu2021neuspike,
  author = {Lin Zhu and Jianing Li and Xiao Wang and Tiejun Huang and Yonghong Tian},
  title = {{NeuSpike-Net: High Speed Video Reconstruction via Bio-inspired Neuromorphic Cameras}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.00240},
}
Figure