Deep Discriminative Spatial and Temporal Network for Efficient Video Deblurring

Jinshan Pan, Boming Xu, Jiangxin Dong, Jianjun Ge, Jinhui Tang

Abstract

Video deblurring is a critical task for restoring clarity to sequences affected by motion blur. Traditional approaches often rely on computationally intensive alignment modules, such as optical flow or deformable convolutions, which limit their real-time applicability. This paper introduces the Deep Discriminative Spatial and Temporal Network (DSTNet), a lightweight framework that achieves state-of-the-art performance without explicit alignment. By leveraging dynamic spatial filters, discriminative temporal fusion, and wavelet-based feature propagation, DSTNet offers a highly efficient solution for high-quality video restoration.

Introduction

The challenge of video deblurring lies in effectively aggregating information across multiple frames to recover sharp details. Most existing methods attempt to align neighboring frames to a reference frame before fusion. However, alignment is inherently difficult when frames are severely blurred, and the associated computational cost (FLOPs) is often prohibitive for mobile or edge devices. DSTNet shifts the paradigm from explicit alignment to discriminative fusion, utilizing adaptive modules to extract and propagate relevant features efficiently.

Methodology

The DSTNet architecture is composed of three primary components designed for efficiency and accuracy:

  1. Channel-wise Gated Dynamic Network (CWGDN): Standard convolutions use static kernels that are content-agnostic. CWGDN instead uses a Multi-Layer Perceptron (MLP) to generate dynamic filters that adapt to the specific spatial characteristics of each frame. This allows the model to capture complex spatial details with significantly fewer parameters than Transformer-based architectures.
  2. Discriminative Temporal Feature Fusion (DTFF): To avoid the pitfalls of inaccurate alignment, DTFF employs a gating mechanism that "discriminates" between useful and noisy features from adjacent frames. This ensures that only high-quality temporal information contributes to the restoration process.
  3. Wavelet-based Feature Propagation (WaveletFP): Long-range temporal information is crucial for consistent deblurring. DSTNet propagates features in the wavelet domain, focusing on low-frequency components. This strategy reduces the accumulation of artifacts and lowers the computational burden compared to full-resolution feature propagation.

Experimental Results

DSTNet was benchmarked against leading models on the GoPro, DVD, and BSD datasets.

Quantitative Performance

On the GoPro dataset, the "Ours-L" version of DSTNet achieved a peak signal-to-noise ratio (PSNR) of 35.05 dB. As shown in Figure 1, DSTNet occupies the upper-left quadrant of the performance-efficiency plot, indicating superior accuracy at a fraction of the computational cost of rivals like BasicVSR++ and Restormer.

Figure 1: PSNR vs. FLOPs comparison. DSTNet (Ours) achieves state-of-the-art accuracy while maintaining significantly lower computational complexity (FLOPs) than competing methods.

Efficiency and Speed

The base DSTNet model operates at approximately 12.5 frames per second (FPS) on an NVIDIA RTX A6000. This represents a 5x to 30x speedup over high-performance models such as RVRT or EDVR. Furthermore, the model is highly compact, with parameter counts as low as 7.45M.

Qualitative Analysis

Visual results demonstrate DSTNet's ability to recover sharp edges and fine textures in dynamic scenes, effectively handling real-world motion blur.

Figure 6: Qualitative deblurring results. DSTNet effectively restores fine details, such as license plate numbers and structural edges, in real-world video sequences.

Conclusion

DSTNet provides a robust and efficient framework for video deblurring by replacing heavy alignment modules with discriminative and dynamic components. Its ability to maintain high accuracy while significantly reducing computational overhead makes it a promising candidate for real-world video processing applications. Future research will focus on improving robustness to abrupt scene transitions through semantic scene understanding.

Summary updated

Abstract

How to effectively explore spatial and temporal information is important for video deblurring. In contrast to existing methods that directly align adjacent frames without discrimination, we develop a deep discriminative spatial and temporal network to facilitate the spatial and temporal feature exploration for better video deblurring. We first develop a channel-wise gated dynamic network to adaptively explore the spatial information. As adjacent frames usually contain different contents, directly stacking features of adjacent frames without discrimination may affect the latent clear frame restoration. Therefore, we develop a simple yet effective discriminative temporal feature fusion module to obtain useful temporal features for latent frame restoration. Moreover, to utilize the information from long-range frames, we develop a wavelet-based feature propagation method that takes the discriminative temporal feature fusion module as the basic unit to effectively propagate main structures from long-range frames for better video deblurring. We show that the proposed method does not require additional alignment methods and performs favorably against state-of-the-art ones on benchmark datasets in terms of accuracy and model complexity.
BibTeX
@inproceedings{pan2023deep,
  author = {Jinshan Pan and Boming Xu and Jiangxin Dong and Jianjun Ge and Jinhui Tang},
  title = {{Deep Discriminative Spatial and Temporal Network for Efficient Video Deblurring}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02125},
}
Figure