Frequency Disentanglement Distillation Image Deblurring Network

Yiming Liu, Jianping Guo, Sen Yang, Ting Liu, Hualing Zhou, Mengzi Liang, Xi Li, Dahong XuSensors 2021 · DOI

TL;DR

This paper introduces the Frequency Disentanglement Distillation Image Deblurring Network (FDDN), a novel encoder-decoder architecture that tackles the challenge of disentangling blur and content information in blind image deblurring. By leveraging frequency domain analysis, FDDN effectively separates low-frequency blur artifacts from high-frequency image details, leading to significantly improved deblurring quality.

What Problem Does This Paper Address?

Blind image deblurring, the task of recovering a sharp image from a blurred one without prior knowledge of the blur kernel, is notoriously difficult. A key challenge is the entanglement of blur information (often residing in low-frequency image components) and content information (typically in high-frequency components). Existing encoder-decoder models struggle to fully separate these, leading to residual blur artifacts and unnatural textures in the restored images. This paper aims to address this entanglement issue by explicitly disentangling blur and content information in the latent space of a deep neural network.

Methods

The core innovation of this paper is the Frequency Disentanglement Distillation Image Deblurring Network (FDDN). FDDN builds upon an encoder-decoder framework but introduces two key novel components:

  • Frequency Split Block (FSB): This block, inspired by OctConv, splits feature channels into low-frequency and high-frequency components. This separation is crucial for isolating blur-related information from content-related information.
  • Frequency Distillation Block (FDB): This block integrates the FSB and a distillation mechanism. It recursively processes the low-frequency features, progressively distilling away blur information while preserving high-frequency content. The FDB is strategically placed at the end of the encoder, operating in the latent space where feature dimensions are smaller and computations are more efficient.

The FDDN architecture also incorporates standard elements like inception down-sampling blocks in the encoder, residual channel attention blocks (RCAB) in the decoder for feature reorganization, and pixel-shuffle convolutions for up-sampling.

Datasets

The paper evaluates FDDN on three datasets:

  • GoPro: A widely used dataset for blind image deblurring, consisting of synthetic blurry and sharp image pairs generated by convolving sharp images with blur kernels.
  • HIDE: A dataset designed for human-aware image deblurring, featuring real-world blurry images with extensive annotations.
  • Karate Dataset: A newly introduced real-world dataset of karate competitions. This dataset is unpaired, meaning it contains only blurred images without corresponding ground truth, making it a more challenging benchmark for evaluating deblurring algorithms in realistic scenarios.

Key Insights and Ideas

  • Frequency Domain as a Disentanglement Tool: The paper's central idea is that the frequency domain provides a natural way to separate blur (low-frequency) from content (high-frequency). This insight is the foundation of their FSB and FDB.
  • Latent Space Operations for Efficiency: Placing the frequency distillation operations in the latent space of the encoder-decoder network significantly reduces computational complexity and memory requirements, as features are smaller in this stage.
  • Recursive Distillation for Refinement: The FDB's recursive nature allows for a step-by-step refinement process, progressively removing blur artifacts and improving the quality of the restored image.
  • Explicit Guidance vs. Implicit Learning: FDDN explicitly guides the network to remove blurry information, unlike some methods that rely solely on training data and loss functions to implicitly learn this.

Key Results

  • Quantitative Performance: FDDN achieves state-of-the-art or competitive results on the GoPro and HIDE datasets, demonstrating strong performance in terms of PSNR and SSIM. For instance, on the GoPro dataset, FDDN achieved a PSNR of 31.42 and SSIM of 0.923, outperforming several existing methods.
  • Efficiency: The model exhibits a significantly smaller model size (8.08 MB on GoPro) compared to some other methods, indicating improved efficiency. Inference time is also competitive.
  • Qualitative Improvements: Visual comparisons show that FDDN effectively removes blur artifacts, recovers fine details, and produces visually pleasing sharp images, even on challenging real-world scenarios like motion blur in the karate dataset.
  • Ablation Studies: Experiments confirm the crucial role of both the frequency split block and the frequency distillation block, with their removal leading to a significant drop in deblurring performance.

Limitations and Future Work

  • Disentanglement is Indirect: The paper acknowledges that FDDN does not directly solve the entanglement of blur and content but rather indirectly achieves it through frequency-based distillation.
  • Future Directions: The authors propose several avenues for future research:
  • Extending FDDN to other image restoration tasks like image deraining and super-resolution.
  • Integrating transformer mechanisms to further enhance image restoration quality.
  • Reducing model parameters to enable real-time deblurring.

Summary updated

Abstract

Due to the blur information and content information entanglement in the blind deblurring task, it is very challenging to directly recover the sharp latent image from the blurred image. Considering that in the high-dimensional feature map, blur information mainly exists in the low-frequency region, and content information exists in the high-frequency region. In this paper, we propose a encoder-decoder model to realize disentanglement from the perspective of frequency, and we named it as frequency disentanglement distillation image deblurring network (FDDN). First, we modified the traditional distillation block by embedding the frequency split block (FSB) in the distillation block to separate the low-frequency and high-frequency region. Second, the modified distillation block, we named frequency distillation block (FDB), can recursively distill the low-frequency feature to disentangle the blurry information from the content information, so as to improve the restored image quality. Furthermore, to reduce the complexity of the network and ensure the high-dimension of the feature map, the frequency distillation block (FDB) is placed on the end of encoder to edit the feature map on the latent space. Quantitative and qualitative experimental evaluations indicate that the FDDN can remove the blur effect and improve the image quality of actual and simulated images.
BibTeX
@inproceedings{liu2021frequency,
  author = {Yiming Liu and Jianping Guo and Sen Yang and Ting Liu and Hualing Zhou and Mengzi Liang and Xi Li and Dahong Xu},
  title = {{Frequency Disentanglement Distillation Image Deblurring Network}},
  booktitle = {Sensors},
  year = {2021},
  doi = {10.3390/s21144702},
}
Figure