Introduction
Motion deblurring remains a critical challenge in computer vision, particularly in high-speed scenarios where traditional frame-based cameras suffer from severe information loss during exposure windows. While deep learning has advanced the field, physical texture loss in extreme blur cases often limits recovery. Neuromorphic "spike cameras" offer a solution by capturing continuous light intensity at ultra-high temporal resolutions (e.g., 20,000 Hz). However, integrating these sensors into practical hardware is difficult due to inevitable spatial and temporal misalignments.
The Alignment Challenge
In real-world deployments, perfectly synchronizing an RGB camera with a spike camera is nearly impossible. Mechanical vibrations, differing focal lengths, and clock drift lead to unknown spatial offsets and temporal synchronization errors. Most existing spike-guided deblurring methods assume "perfect alignment," causing them to fail when faced with the "messy" data of actual hardware rigs. This paper addresses these real-world hurdles by introducing a framework that functions without precise calibration.
Proposed Method: UaSDN
The authors propose the Unaligned Spike-guided Deblur Net (UaSDN), a three-stage architecture designed to handle unknown offsets:
- Coarse Restoration: A simplified NAFNet processes the blurred RGB image to recover initial textures and remove primary blur.
- Bi-Directional Modal Alignment (BiDMA): This core innovation uses deformable convolutions to learn a spatial mapping between spike and image features. By "warping" modalities into a shared feature space, it eliminates the need for manual calibration.
- Flow-guided Dense Fusion: The model estimates optical flow from the high-speed spike stream to guide a multi-scale fusion network, injecting sharp, high-frequency details into the final RGB output.
Experimental Results
The authors evaluated UaSDN using both simulated datasets (X4K1000FPS, REDS) with introduced perturbations and real-world data from a custom hybrid camera rig.
Quantitative Performance
On the X4K1000FPS dataset, UaSDN achieved a PSNR of 35.78 dB, representing a 1.31 dB improvement over the previous state-of-the-art spike-based method (SpkDeblurNet). It also outperformed standard image-only methods by more than 1.6 dB.
Qualitative and Real-World Robustness
Visual evaluations demonstrate that UaSDN successfully recovers fine details, such as legible text on moving license plates and sharp edges on high-speed objects, where competing methods produce smeared results. Crucially, the model proved robust to spatial offsets of up to 10% and temporal offsets of up to 25%, making it highly effective for real-world hardware applications where "ideal data" is rarely available.
Conclusion
UaSDN marks a significant step toward practical neuromorphic-enhanced imaging. By explicitly addressing the reality of unaligned sensors, it enables high-quality motion deblurring in dynamic environments. Future work may involve optimizing the architecture for real-time edge deployment and extending the unaligned alignment strategy to other tasks like high-dynamic-range (HDR) reconstruction or super-resolution.
Summary updated