Exploring Temporal Frequency Spectrum in Deep Video Deblurring

Qi Zhu, Man Zhou, Naishan Zheng, Chongyi Li, Jie Huang, Feng Zhao

The Challenge of Temporal Modeling

Video deblurring is the difficult task of recovering sharp latent frames from a sequence of blurry ones. The core challenge lies in temporal modeling: how do you align and aggregate information from neighboring frames when those frames are smeared by motion?

Existing deep learning methods typically operate in the spatial domain, using tools like deformable convolutions or optical flow for alignment. However, when blur is severe, high-frequency details—the very features needed for accurate alignment—are lost. This creates a "catch-22" where you need a sharp image to align frames, but you need aligned frames to recover a sharp image. This paper argues that the potential of the frequency domain (Fourier space) has been largely overlooked in solving this bottleneck.

Methods: A Frequency-Temporal Solution

The authors propose a novel Fourier-based pipeline that injects spectral information into the standard deblurring stages of extraction, alignment, and fusion.

  1. Spatial-Frequency Feature Extraction (SFE): This block uses the Discrete Fourier Transform (DFT) to process the amplitude and phase of features separately. This allows the model to explicitly model blur degradation where it is most visible.
  2. Spectrum Prior-guided Alignment (SPA): To fix the alignment issue, the authors use the frequency spectrum to modulate spatial offsets. Since the frequency domain "enlarges" the unpredictable changes caused by motion blur, it provides a more robust signal for aligning frames even when spatial details are smeared.
  3. Temporal Energy Attention (TEA): Based on the observation that sharper frames have higher "spectral energy" (L2 norm of the spectrum), this module uses energy as a global sharpness indicator. It assigns higher weights to clearer patches from neighboring frames during aggregation.
Illustration of how spectral energy (L2 norm) decreases as motion blur increases, providing a mathematical proxy for frame sharpness.

Key Insights

The "Separability" Discovery

Using t-SNE clusters, the authors demonstrate that blurry and sharp frames are "tangled" and hard to distinguish in the spatial domain. However, in the frequency domain, they become linearly separable. This discovery justifies moving the computation into Fourier space to simplify the deblurring task.

Energy as a Proxy for Sharpness

A fundamental physical prior used here is that blur acts as a low-pass filter, decaying the energy of the spectrum. By monitoring the L2 norm of the spectrum, the model can mathematically "sense" which frames in a sequence are the sharpest without needing a reference.

t-SNE visualization showing the separability of sharp and blurry frames in the frequency domain (top) vs. the spatial domain (bottom), along with spectral energy trends over time.

Directional Priors

The paper demonstrates that motion blur creates specific "strip" patterns in the frequency spectrum that are perpendicular to the direction of motion. This makes the frequency domain a natural fit for estimating motion parameters and guiding alignment.

Results and Performance

The proposed method achieves state-of-the-art performance on the DVD (Deep Video Deblurring) dataset:

  • Quantitative: It reached a PSNR of 33.25 dB, outperforming the previous best method (RTA) by 0.33 dB and significantly beating the popular EDVR (31.91 dB).
  • Efficiency: A "small" version of the model (4.04M parameters) still outperforms many much larger models, proving that frequency-domain priors are more efficient than "brute-force" spatial convolutions.
  • Qualitative: Visual results show that the method is particularly effective at restoring fine textures, such as text on signs or license plates, which often remain distorted in purely spatial methods.

Limitations and Future Work

While the model is parameter-efficient, performing multiple DFT and IDFT operations can be computationally intensive for real-time applications on mobile hardware. Additionally, the current approach uses a simplified motion blur model that does not fully account for depth variations in a scene. The authors suggest that exploring more complex, non-uniform blur kernels within the frequency domain could further improve results in highly dynamic environments.

Summary updated

Abstract

Video deblurring aims to restore the latent video frames from their blurred counterparts. Despite the remarkable progress, most promising video deblurring methods only investigate the temporal priors in the spatial domain and rarely explore their its potential in the frequency domain. In this paper, we revisit the blurred sequence in the Fourier space and figure out some intrinsic frequency-temporal priors that imply the temporal blur degradation can be accessibly decoupled in the potential frequency domain. Based on these priors, we propose a novel Fourier-based frequency-temporal video deblurring solution, where the core design accommodates the temporal spectrum to a popular video deblurring pipeline of feature extraction, alignment, aggregation, and optimization. Specifically, we design a Spectrum Prior-guided Alignment module by leveraging enlarged blur information in the potential spectrum to mitigate the blur effects on the alignment. Then, Temporal Energy prior-driven Aggregation is implemented to replenish the original local features by estimating the temporal spectrum energy as the global sharpness guidance. In addition, the customized frequency loss is devised to optimize the proposed method for decent spectral distribution. Extensive experiments demonstrate that our model performs favorably against other state-of-the-art methods, thus confirming the effectiveness of frequency-temporal prior modeling.
BibTeX
@inproceedings{zhu2023exploring,
  author = {Qi Zhu and Man Zhou and Naishan Zheng and Chongyi Li and Jie Huang and Feng Zhao},
  title = {{Exploring Temporal Frequency Spectrum in Deep Video Deblurring}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
  doi = {10.1109/iccv51070.2023.01142},
}
Figure