ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and Interpolation

Akash Gupta, Abhishek Aich, Amit K. Roy-Chowdhury

Abstract

Existing works address the problem of generating high frame-rate sharp videos by separately learning the frame deblurring and frame interpolation modules. Most of these approaches have a strong prior assumption that all the input frames are blurry whereas in a real-world setting, the quality of frames varies. Moreover, such approaches are trained to perform either of the two tasks - deblurring or interpolation - in isolation, while many practical situations call for both. Different from these works, we address a more realistic problem of high frame-rate sharp video synthesis with no prior assumption that input is always blurry. We introduce a novel architecture, Adaptive Latent Attention Network (ALANET), which synthesizes sharp high frame-rate videos with no prior knowledge of input frames being blurry or not, thereby performing the task of both deblurring and interpolation. We hypothesize that information from the latent representation of the consecutive frames can be utilized to generate optimized representations for both frame deblurring and frame interpolation. Specifically, we employ combination of self-attention and cross-attention module between consecutive frames in the latent space to generate optimized representation for each frame. The optimized representation learnt using these attention modules help the model to generate and interpolate sharp frames. Extensive experiments on standard datasets demonstrate that our method performs favorably against various state-of-the-art approaches, even though we tackle a much more difficult problem.

Introduction

In mobile videography, motion blur and low frame rates often degrade video quality. Traditionally, deblurring and interpolation have been treated as separate tasks. However, this "cascade" approach suffers from error propagation and relies on the unrealistic assumption that all input frames are uniformly blurry. In real-world scenarios, video quality fluctuates, with some frames remaining sharp while others are blurred.

Overview of the ALANET framework for joint deblurring and interpolation.

ALANET (Adaptive Latent Attention Network) addresses the Joint Video Deblurring and Interpolation (JVDI) problem. It transforms low-frame-rate, variable-quality video into high-frame-rate, sharp video without prior knowledge of which frames are blurry.

Methodology

ALANET utilizes a U-Net-style architecture consisting of an Encoder ($\mathcal{E}$), an Adaptive Latent Attention module ($\mathcal{M}$), and a Generator ($\mathcal{G}$).

Detailed architecture of ALANET showing the encoder, attention module, and generator.

Latent Space Representation

Instead of operating directly in pixel space, ALANET projects frames into a compressed, feature-rich latent space. This approach is computationally efficient and allows the network to focus on high-level semantic features rather than raw pixel noise.

Adaptive Latent Attention

The core of ALANET is the attention module ($\mathcal{M}$), which employs two mechanisms:

  • Self-Attention ($M_S$): Identifies critical features within a single frame to assist in its reconstruction.
  • Cross-Attention ($M_C$): Operates across a temporal window of neighboring frames. It allows the model to adaptively "borrow" sharp details from adjacent frames to compensate for blur in the current frame.

This joint optimization ensures that the latent space is robust for both deblurring original frames and interpolating new ones simultaneously.

Experiments and Results

The model was evaluated on the Adobe240 dataset (synthetic blur) and the YouTube240 dataset (real-world footage).

Quantitative Performance

On Adobe240, ALANET achieved a 1.04dB improvement in PSNR for deblurring over the BIN baseline. For the joint task, it demonstrated a relative improvement of 2.3% in PSNR and 1.04% in SSIM (Structural Similarity Index).

Qualitative Analysis

Visually, ALANET effectively removes "ghosting" artifacts and preserves sharp edges in high-motion scenes where previous methods often failed. Its performance on the YouTube240 dataset confirms its ability to generalize to "in the wild" footage without additional fine-tuning, proving it has learned the underlying physics of motion and blur.

Conclusion

ALANET represents a significant advancement in video restoration by jointly tackling deblurring and interpolation in a shared latent space. While limited by a fixed temporal window and challenges with extreme motion, the adaptive latent attention mechanism provides a foundation for future universal video enhancement networks, including tasks like super-resolution and denoising.

Summary updated

Abstract

Existing works address the problem of generating high frame-rate sharp videos by separately learning the frame deblurring and frame interpolation modules. Most of these approaches have a strong prior assumption that all the input frames are blurry whereas in a real-world setting, the quality of frames varies. Moreover, such approaches are trained to perform either of the two tasks - deblurring or interpolation - in isolation, while many practical situations call for both. Different from these works, we address a more realistic problem of high frame-rate sharp video synthesis with no prior assumption that input is always blurry. We introduce a novel architecture, Adaptive Latent Attention Network (ALANET), which synthesizes sharp high frame-rate videos with no prior knowledge of input frames being blurry or not, thereby performing the task of both deblurring and interpolation. We hypothesize that information from the latent representation of the consecutive frames can be utilized to generate optimized representations for both frame deblurring and frame interpolation. Specifically, we employ combination of self-attention and cross-attention module between consecutive frames in the latent space to generate optimized representation for each frame. The optimized representation learnt using these attention modules help the model to generate and interpolate sharp frames. Extensive experiments on standard datasets demonstrate that our method performs favorably against various state-of-the-art approaches, even though we tackle a much more difficult problem.
BibTeX
@inproceedings{gupta2020alanet,
  author = {Akash Gupta and Abhishek Aich and Amit K. Roy-Chowdhury},
  title = {{ALANET: Adaptive Latent Attention Network forJoint Video Deblurring and Interpolation}},
  booktitle = {ACM Multimedia},
  year = {2020},
  url = {https://openreview.net/forum?id=YlAOPG98YNj},
}
Figure