Adversarial Spatio-Temporal Learning for Video Deblurring

Kaihao Zhang, Wenhan Luo, Yiran Zhong, Lin Ma, Wei Liu, Hongdong Li

Abstract

Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and temporal domain (i.e., neighboring frames), and 2) how to restore sharp image details w.r.t. the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a DeBLuRring Network (DBLRNet) for spatial-temporal learning by applying a modified 3D convolution to both spatial and temporal domains. Our DBLRNet is able to capture jointly spatial and temporal information encoded in neighboring frames, which directly contributes to improved video deblur performance. To tackle the second challenge, we leverage the developed DBLRNet as a generator in the GAN (generative adversarial network) architecture, and employ a content loss in addition to an adversarial loss for efficient adversarial training. The developed network, which we name as DeBLuRring Generative Adversarial Network (DBLRGAN), is tested on two standard benchmarks and achieves the state-of-the-art performance.

Introduction

Video deblurring is significantly more complex than single-image deblurring. When a camera shakes or an object moves quickly, the resulting blur is a spatio-temporal phenomenon. Existing research often struggles with two major gaps: modeling complex dynamics across neighboring frames and avoiding the "PSNR trap." Traditionally, models optimized for Peak Signal-to-Noise Ratio (PSNR) tend to produce "safe," blurry-looking images because they minimize pixel-wise error rather than maximizing visual sharpness. Humans generally prefer slightly noisy but sharp images over smooth, blurry ones.

Methods: DBLRGAN Architecture

The proposed solution consists of two primary components: the DBLRNet generator and the DBLRGAN adversarial framework.

DBLRNet (The Generator)

Instead of processing frames individually, DBLRNet takes a stack of five consecutive frames. It utilizes 3D (volumetric) convolutions within a deep residual architecture. Unlike 2D convolutions that only slide across the height and width of an image, 3D kernels slide across height, width, and time. This allows the network to jointly learn features from the spatial plane and the temporal evolution of the blur. The model operates in the YCbCr color space, specifically deblurring the Y (luminance) channel to preserve salient structural information.

Adversarial Framework

To overcome the limitations of pixel-wise loss, DBLRNet is integrated into a Generative Adversarial Network (GAN). The generator (DBLRNet) attempts to create a sharp version of the middle frame, while a discriminator tries to distinguish between the restored image and a ground-truth sharp image. The training process uses a weighted combination of Content Loss (MSE) and Adversarial Loss, pushing the model to produce fine-grained textures and sharp edges.

The DBLRGAN framework, showing the DBLRNet generator integrated with a discriminator and dual loss functions.

Key Insights and Results

The model was evaluated on the VideoDeblurring and Blurred KITTI datasets. Several key findings emerged:

  • 3D Convolutions vs. 2D Stacking: Using 3D convolutions to preserve the temporal axis led to a 3-5% improvement in PSNR compared to simply stacking frames as RGB channels.
  • Optimal Temporal Window: Testing sequences from 3 to 11 frames revealed that 5 frames provided the best balance between context and motion complexity.
  • Alignment-Free Efficiency: DBLRGAN works on raw, unaligned frames, making it up to 20 times faster than traditional methods requiring optical flow or image alignment.

Quantitatively, DBLRGAN achieved an average PSNR of 33.19 dB on the VideoDeblurring benchmark, outperforming the previous state-of-the-art (DBN) by over 3 dB. Qualitatively, the GAN-trained version produced noticeably sharper edges and more legible text compared to models trained only on MSE.

Qualitative comparison showing DBLRGAN's ability to restore sharp details compared to other state-of-the-art methods.

Conclusion

DBLRGAN introduces an effective way to handle the spatio-temporal nature of video blur using 3D convolutions and adversarial training. While the model excels at sharpening individual frames, future work will focus on improving temporal consistency to eliminate minor "jumping artifacts" or jitter during high-speed playback. By avoiding expensive alignment steps, DBLRGAN offers a high-performance, efficient path for real-world video restoration.

Summary updated

Abstract

Camera shake or target movement often leads to undesired blur effects in videos captured by a hand-held camera. Despite significant efforts having been devoted to video-deblur research, two major challenges remain: 1) how to model the spatio-temporal characteristics across both the spatial domain (i.e., image plane) and temporal domain (i.e., neighboring frames), and 2) how to restore sharp image details w.r.t. the conventionally adopted metric of pixel-wise errors. In this paper, to address the first challenge, we propose a DeBLuRring Network (DBLRNet) for spatial-temporal learning by applying a modified 3D convolution to both spatial and temporal domains. Our DBLRNet is able to capture jointly spatial and temporal information encoded in neighboring frames, which directly contributes to improved video deblur performance. To tackle the second challenge, we leverage the developed DBLRNet as a generator in the GAN (generative adversarial network) architecture, and employ a content loss in addition to an adversarial loss for efficient adversarial training. The developed network, which we name as DeBLuRring Generative Adversarial Network (DBLRGAN), is tested on two standard benchmarks and achieves the state-of-the-art performance.
BibTeX
@article{zhang2019adversarial,
  author = {Kaihao Zhang and Wenhan Luo and Yiran Zhong and Lin Ma and Wei Liu and Hongdong Li},
  title = {{Adversarial Spatio-Temporal Learning for Video Deblurring}},
  journal = {IEEE Trans. Image Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=HIQZNEGpDxq},
}
Figure