Abstract
NeuSpike-Net is a deep learning framework that mimics the human visual system by fusing data from two distinct neuromorphic sensors: the Dynamic Vision Sensor (DVS) and the Spike Camera. By combining the high-speed motion sensitivity of DVS with the fine-grained texture sampling of Spike Cameras, the proposed architecture achieves high-quality, high-dynamic-range (HDR) video reconstruction that surpasses the capabilities of either sensor in isolation. This paper details the dual-path encoder design and the novel Feature Fusion Module (FFM) that enables robust performance in extreme conditions such as ultra-high-speed motion and low-light environments.
Introduction
Capturing high-speed scenes with high dynamic range is a major challenge for conventional frame-based cameras, which suffer from motion blur and over/under-exposure. Bio-inspired neuromorphic sensors offer a solution, but the field has historically been split between two distinct sensor types:
- Dynamic Vision Sensors (DVS): Similar to the periphery of the human retina, these detect temporal changes in light. They are excellent for motion and HDR but struggle to reconstruct static textures.
- Spike Cameras: Similar to the fovea, these accumulate light intensity to fire spikes. They capture rich texture but are highly sensitive to noise and have a limited dynamic range.
NeuSpike-Net addresses the gap of how to effectively fuse these two heterogeneous, asynchronous data streams to create a "super-sensor" capable of reconstructing high-fidelity video.
Methods
The authors propose NeuSpike-Net, a U-Net-style architecture featuring a dual-path encoder and a shared decoder.
Adaptive Data Representation
To handle the asynchronous nature of the data, Spike data is represented using a temporal Inter-Spike Interval (ISI) map, while Event data is processed through an "event integration" model inspired by the integrate-and-fire neuron.
Dual-Path Encoding and Fusion
The architecture utilizes two specialized paths:
- Motion Path: Processes DVS events to capture high-speed movement and HDR information.
- Texture Path: Processes Spike data to extract fine spatial details.
The core innovation is the Feature Fusion Module (FFM). Instead of simple concatenation, the FFM uses Channel and Spatial Attention to learn dynamic weights for features from both paths. For instance, the network learns to rely more on DVS in dark regions where Spike noise is high, or on Spike data when the scene is static and DVS is "blind."
Experimental Evaluation
The framework was evaluated using both simulated and real-world datasets. The simulated dataset was derived from KITTI, upsampled to 22,500 FPS to provide ground truth. The real-world dataset was captured using a custom hybrid rig aligning a Vidar spike camera and a DAVIS 346 event camera.
Key Results
- Quantitative Superiority: On simulated "Complex" scenes (high speed + low light), NeuSpike-Net achieved a PSNR of 27.40 and SSIM of 0.8510, significantly outperforming DVS-only methods like E2VID (PSNR 12.85) and Spike-only methods like SNM (PSNR 26.44).
- Real-World Performance: In tests with a fan spinning at 2600 RPM, NeuSpike-Net was the only method able to reconstruct clear, legible letters on the fan blades without significant artifacts.
- HDR Handling: The system successfully reconstructed details in scenes where traditional sensors would have seen only black shadows or blown-out highlights.
Conclusion
NeuSpike-Net demonstrates that the DVS and Spike camera are naturally complementary, much like the human eye's peripheral and foveal vision. While the system achieves state-of-the-art results, future work will address challenges in sub-pixel hardware alignment and computational complexity to enable deployment on edge devices like drones or autonomous vehicles.
Summary updated