Abstract
Spike cameras offer unprecedented temporal resolution for capturing high-speed scenes but are currently limited by low spatial resolution and significant temporal noise, or "fluctuations." We present Spk2SRImgNet, a deep learning framework designed to denoise and super-resolve spike streams into high-quality, high-resolution (HR) images. The core of our approach is the Motion Aligned Collaborative Filtering (MACF) module, which leverages temporal redundancy and transform-domain filtering to overcome spike timing instabilities. Experimental results demonstrate that Spk2SRImgNet outperforms existing state-of-the-art methods in both reconstruction quality and computational efficiency.
Introduction
Conventional frame-based cameras are fundamentally limited by their exposure windows; fast-moving objects often result in motion blur. Neuromorphic spike cameras address this by utilizing a sensor array where each pixel fires a binary spike independently upon accumulating a specific amount of light. This allows for an effective capture rate of 40,000 Hz.
Despite this temporal advantage, spike cameras face two primary challenges: hardware constraints that limit spatial resolution and "spike fluctuations." These fluctuations—caused by photon noise, thermal noise, and quantization errors—result in inconsistent spike intervals even under constant light intensity. These hurdles make it difficult to align data across time for effective super-resolution.
Methodology
Spk2SRImgNet processes raw spike streams through a multi-stage pipeline. The stream is first partitioned and passed through Spike Intensity Reconstruction (SIR) and Encoder modules to extract initial features. The defining innovation of this framework is the Motion Aligned Collaborative Filtering (MACF) module.
Inspired by classical image restoration techniques like BM3D, the MACF module performs four key operations:
- Offset Estimation & Warping: Deformable convolutions align features from neighboring frames to a central key frame, compensating for scene motion.
- Channel Rearrangement (CR): Features from different time steps are reorganized to group similar channels together.
- Transform Domain Filtering: These grouped features are filtered in a transform domain to exploit representation sparsity, effectively separating the consistent scene signal from random temporal noise.
- Aggregation: Refined features are warped back to their original positions and aggregated to produce a stable, denoised feature set for final HR reconstruction.
Experimental Results
Due to the difficulty of capturing simultaneous HR ground truth with real spike hardware, the model was trained on synthesized datasets derived from the REDS and Adobe240 video benchmarks. The model was then validated on real-world data captured at 250 × 400 resolution.
Quantitative Performance
On the REDS-based x4 super-resolution task, Spk2SRImgNet achieved a PSNR of 29.50 dB, surpassing the previous state-of-the-art (29.22 dB). Furthermore, the model is highly efficient; it completes a 4x upscaling task in 0.21 seconds, roughly five times faster than SpikeSR-Net (1.07 seconds).
Qualitative Analysis
Visual evaluations on real-world high-speed subjects, such as rotating fans and moving clocks, show that Spk2SRImgNet effectively eliminates "ghosting" artifacts and restores sharp structural details that previous methods fail to capture.
Conclusion
Spk2SRImgNet successfully bridges the gap between high-speed neuromorphic sensing and high-quality visual reconstruction. By integrating classical collaborative filtering logic into a modern deep learning architecture, the framework effectively addresses the temporal instability inherent in spike cameras. Future work will explore incorporating spatial non-local similarity and developing more robust real-world benchmarking rigs to further push the boundaries of spike-based super-resolution.
Summary updated