Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment

Jiyuan Zhang, Shiyan Chen, Yajing Zheng, Zhaofei Yu, Tiejun Huang

Introduction

Motion deblurring remains a critical challenge in computer vision, particularly in high-speed scenarios where traditional frame-based cameras suffer from severe information loss during exposure windows. While deep learning has advanced the field, physical texture loss in extreme blur cases often limits recovery. Neuromorphic "spike cameras" offer a solution by capturing continuous light intensity at ultra-high temporal resolutions (e.g., 20,000 Hz). However, integrating these sensors into practical hardware is difficult due to inevitable spatial and temporal misalignments.

Spatiotemporal misalignment challenges between RGB and spike cameras.

The Alignment Challenge

In real-world deployments, perfectly synchronizing an RGB camera with a spike camera is nearly impossible. Mechanical vibrations, differing focal lengths, and clock drift lead to unknown spatial offsets and temporal synchronization errors. Most existing spike-guided deblurring methods assume "perfect alignment," causing them to fail when faced with the "messy" data of actual hardware rigs. This paper addresses these real-world hurdles by introducing a framework that functions without precise calibration.

Proposed Method: UaSDN

The authors propose the Unaligned Spike-guided Deblur Net (UaSDN), a three-stage architecture designed to handle unknown offsets:

  1. Coarse Restoration: A simplified NAFNet processes the blurred RGB image to recover initial textures and remove primary blur.
  2. Bi-Directional Modal Alignment (BiDMA): This core innovation uses deformable convolutions to learn a spatial mapping between spike and image features. By "warping" modalities into a shared feature space, it eliminates the need for manual calibration.
  3. Flow-guided Dense Fusion: The model estimates optical flow from the high-speed spike stream to guide a multi-scale fusion network, injecting sharp, high-frequency details into the final RGB output.

Experimental Results

The authors evaluated UaSDN using both simulated datasets (X4K1000FPS, REDS) with introduced perturbations and real-world data from a custom hybrid camera rig.

Comparison of deblurring performance across different alignment scenarios and the UaSDN architecture concept.

Quantitative Performance

On the X4K1000FPS dataset, UaSDN achieved a PSNR of 35.78 dB, representing a 1.31 dB improvement over the previous state-of-the-art spike-based method (SpkDeblurNet). It also outperformed standard image-only methods by more than 1.6 dB.

Qualitative and Real-World Robustness

Visual evaluations demonstrate that UaSDN successfully recovers fine details, such as legible text on moving license plates and sharp edges on high-speed objects, where competing methods produce smeared results. Crucially, the model proved robust to spatial offsets of up to 10% and temporal offsets of up to 25%, making it highly effective for real-world hardware applications where "ideal data" is rarely available.

Conclusion

UaSDN marks a significant step toward practical neuromorphic-enhanced imaging. By explicitly addressing the reality of unaligned sensors, it enables high-quality motion deblurring in dynamic environments. Future work may involve optimizing the architecture for real-time edge deployment and extending the unaligned alignment strategy to other tasks like high-dynamic-range (HDR) reconstruction or super-resolution.

Summary updated

Abstract

The traditional frame-based cameras that rely on exposure windows for imaging experience motion blur in high-speed scenarios. Frame-based deblurring methods lack reliable motion cues to restore sharp images under extreme blur conditions. The spike camera is a novel neuromorphic visual sensor that outputs spike streams with ultra-high temporal resolution. It can supplement the temporal information lost in traditional cameras and guide motion deblurring. However, in real-world scenarios, aligning discrete RGB images and continuous spike streams along both temporal and spatial axes is challenging due to the complexity of calibrating their coordinates, device displacements in vibrations, and time deviations. Misalignment of pixels leads to severe degradation of deblurring. We introduce the first framework for spike-guided motion deblurring without knowing the spatiotemporal alignment between spikes and images. To address the problem, we first propose a novel three-stage network containing a basic deblurring net, a carefully designed bi-directional deformable aligning module, and a flow-based multi-scale fusion net. Experimental results demonstrate that our approach can effectively guide the image deblurring with unknown alignment, surpassing the performance of other methods. Public project page: https://github.com/Leozhangjiyuan/UaSDN.
BibTeX
@inproceedings{zhang2024spike,
  author = {Jiyuan Zhang and Shiyan Chen and Yajing Zheng and Zhaofei Yu and Tiejun Huang},
  title = {{Spike-guided Motion Deblurring with Unknown Modal Spatiotemporal Alignment}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.02366},
}
Figure