The Challenge of Temporal Modeling
Video deblurring is the difficult task of recovering sharp latent frames from a sequence of blurry ones. The core challenge lies in temporal modeling: how do you align and aggregate information from neighboring frames when those frames are smeared by motion?
Existing deep learning methods typically operate in the spatial domain, using tools like deformable convolutions or optical flow for alignment. However, when blur is severe, high-frequency details—the very features needed for accurate alignment—are lost. This creates a "catch-22" where you need a sharp image to align frames, but you need aligned frames to recover a sharp image. This paper argues that the potential of the frequency domain (Fourier space) has been largely overlooked in solving this bottleneck.
Methods: A Frequency-Temporal Solution
The authors propose a novel Fourier-based pipeline that injects spectral information into the standard deblurring stages of extraction, alignment, and fusion.
- Spatial-Frequency Feature Extraction (SFE): This block uses the Discrete Fourier Transform (DFT) to process the amplitude and phase of features separately. This allows the model to explicitly model blur degradation where it is most visible.
- Spectrum Prior-guided Alignment (SPA): To fix the alignment issue, the authors use the frequency spectrum to modulate spatial offsets. Since the frequency domain "enlarges" the unpredictable changes caused by motion blur, it provides a more robust signal for aligning frames even when spatial details are smeared.
- Temporal Energy Attention (TEA): Based on the observation that sharper frames have higher "spectral energy" (L2 norm of the spectrum), this module uses energy as a global sharpness indicator. It assigns higher weights to clearer patches from neighboring frames during aggregation.
Key Insights
The "Separability" Discovery
Using t-SNE clusters, the authors demonstrate that blurry and sharp frames are "tangled" and hard to distinguish in the spatial domain. However, in the frequency domain, they become linearly separable. This discovery justifies moving the computation into Fourier space to simplify the deblurring task.
Energy as a Proxy for Sharpness
A fundamental physical prior used here is that blur acts as a low-pass filter, decaying the energy of the spectrum. By monitoring the L2 norm of the spectrum, the model can mathematically "sense" which frames in a sequence are the sharpest without needing a reference.
Directional Priors
The paper demonstrates that motion blur creates specific "strip" patterns in the frequency spectrum that are perpendicular to the direction of motion. This makes the frequency domain a natural fit for estimating motion parameters and guiding alignment.
Results and Performance
The proposed method achieves state-of-the-art performance on the DVD (Deep Video Deblurring) dataset:
- Quantitative: It reached a PSNR of 33.25 dB, outperforming the previous best method (RTA) by 0.33 dB and significantly beating the popular EDVR (31.91 dB).
- Efficiency: A "small" version of the model (4.04M parameters) still outperforms many much larger models, proving that frequency-domain priors are more efficient than "brute-force" spatial convolutions.
- Qualitative: Visual results show that the method is particularly effective at restoring fine textures, such as text on signs or license plates, which often remain distorted in purely spatial methods.
Limitations and Future Work
While the model is parameter-efficient, performing multiple DFT and IDFT operations can be computationally intensive for real-time applications on mobile hardware. Additionally, the current approach uses a simplified motion blur model that does not fully account for depth variations in a scene. The authors suggest that exploring more complex, non-uniform blur kernels within the frequency domain could further improve results in highly dynamic environments.
Summary updated