Take a Prior from Other Tasks for Severe Blur Removal

Pei Wang, Danna Xue, Yu Zhu, Jinqiu Sun, Qingsen Yan, Sung-eui Yoon, Yanning Zhang

TL;DR

Severe image blur destroys both fine structural details and semantic meaning, making image restoration incredibly challenging. This paper introduces a novel framework that extracts "sharp semantic priors" from high-level vision tasks (like image classification) via knowledge distillation to guide and significantly boost the performance of existing deblurring models.

What Problem Does This Paper Address?

Image deblurring is a classic, ill-posed computer vision problem. When blur is severe—caused by rapid camera shake or fast-moving objects—standard deblurring networks struggle. This is because severe blur causes a massive loss of local detail and introduces semantic ambiguity; it becomes difficult for a network to distinguish object boundaries or even identify what the objects are.

While some specialized methods use semantic segmentation maps to help deblur human faces, this approach fails in general natural scenes. Natural scenes are highly complex, and standard segmentation maps are often too coarse—they provide global category labels but discard the fine textures and local structures essential for high-quality image restoration. The authors aim to bridge this gap by learning a comprehensive prior that captures both global context and sharp local details for general scene deblurring.

Methods

The authors propose a semantic prior-guided framework consisting of two main branches:

  1. Semantic Prior Learning (SPL) Branch: This branch uses a teacher-student knowledge distillation strategy. A "teacher" network (a high-level classification model pre-trained on sharp images) generates clean semantic features. A "student" network takes the blurry image and learns to mimic these sharp teacher features. To achieve this, the authors use:
  2. Hierarchical Context Loss (HCL): An enhanced $L_2$ loss that minimizes feature distances across multiple resolutions to maintain local structural scale.
  3. Cross-Level Connection (CLC) Module: A module that pulls abstract, deep-level semantic features into shallower layers of the student network, enhancing its ability to resolve semantic confusion early on.
  4. Image Deblurring (IDe) Branch: This is the core restoration network (e.g., a standard UNet or other state-of-the-art architectures). The learned priors are integrated into this branch via a Semantic Prior Embedding (SPE) layer, which consists of:
  5. Multi-Level Aggregate (MLA) Module: Merges multi-scale priors from different feature dimensions.
  6. Semantic Attention Transformation (SAT) Module: Dynamically applies scale and shift modulations to the deblurring features, ensuring different image regions receive appropriate weights based on their semantic context and blur severity.

Datasets

The framework was trained and evaluated on three widely recognized datasets:

  • GoPro Dataset: A synthetic dataset containing 2,103 training and 1,111 testing pairs exhibiting dynamic motion blur.
  • HIDE Dataset: A synthetic dataset specifically tailored for human-aware motion blur, featuring more severe degradation due to frame aggregation.
  • RealBlur Dataset (RealBlur-J subset): A real-world dataset consisting of 3,758 training and 980 testing images captured under real-world blurry conditions.

Key Insights and Ideas

  • Classification Over Segmentation: Interestingly, features derived from image classification models provide better structural and contextual guidance for general scene deblurring than segmentation maps. Segmentation focuses heavily on edges and shapes, whereas classification features preserve richer internal object contexts and textures.
  • Feature-Level vs. Image-Level Priors: Instead of feeding a final discrete map (like a segmentation mask) into the deblurring network, utilizing intermediate, continuous feature representations allows the model to retain a powerful blend of abstract semantics and sharp local textures.
  • Privileged Information via Distillation: By utilizing sharp images during training to supervise the student network's interpretation of blurry images, the model effectively learns to "imagine" clean structures even when the input is heavily degraded.

Key Results

The proposed semantic prior (SP) acts as a modular booster that can be plugged into various mainstream architectures, consistently yielding quantitative and qualitative improvements:

  • UNet Boost: On the GoPro dataset, adding the semantic prior (UNet+SP) improved the baseline PSNR by +0.77 dB (from 30.29 dB to 31.06 dB).
  • State-of-the-Art Enhancements: Integrating the prior into advanced architectures led to significant performance jumps, such as a +0.98 dB improvement for MPRNet and a +0.24 dB improvement for MSSNet.
  • Visual Quality: Qualitatively, models equipped with the semantic prior recovered significantly sharper text, cleaner facial features, and fewer structural artifacts in severely blurred zones compared to baseline models.

Limitations and Future Work

Despite its success, the authors note two primary limitations:

  • Out-of-Plane Motion Blur: The method struggles with complex 3D or out-of-plane motion (such as hands shaking toward or away from the camera) because rapid depth changes violate the assumptions of the current feature space.
  • Tiny Structures: When blurred regions contain exceptionally small details (like tiny text characters), the classification model's categories cannot adequately cover or resolve them, leaving a performance gap between the restored image and the ground truth.

Future work will focus on addressing out-of-plane motion blur and exploring new semantic techniques specifically optimized for tiny object deblurring.

Summary updated

Abstract

Recovering clear structures from severely blurry inputs is a challenging problem due to the large movements between the camera and the scene. Although some works apply segmentation maps on human face images for deblurring, they cannot handle natural scenes because objects and degradation are more complex, and inaccurate segmentation maps lead to a loss of details. For general scene deblurring, the feature space of the blurry image and corresponding sharp image under the high-level vision task is closer, which inspires us to rely on other tasks (e.g. classification) to learn a comprehensive prior in severe blur removal cases. We propose a cross-level feature learning strategy based on knowledge distillation to learn the priors, which include global contexts and sharp local structures for recovering potential details. In addition, we propose a semantic prior embedding layer with multi-level aggregation and semantic attention transformation to integrate the priors effectively. We introduce the proposed priors to various models, including the UNet and other mainstream deblurring baselines, leading to better performance on severe blur removal. Extensive experiments on natural image deblurring benchmarks and real-world images, such as GoPro and RealBlur datasets, demonstrate our method's effectiveness and generalization ability.
BibTeX
@article{wang2024take,
  author = {Pei Wang and Danna Xue and Yu Zhu and Jinqiu Sun and Qingsen Yan and Sung-eui Yoon and Yanning Zhang},
  title = {{Take a Prior from Other Tasks for Severe Blur Removal}},
  journal = {Comput. Vis. Image Underst.},
  year = {2024},
  url = {https://openreview.net/forum?id=FaNKvELjQn},
}
Figure