TL;DR
Severe image blur destroys both fine structural details and semantic meaning, making image restoration incredibly challenging. This paper introduces a novel framework that extracts "sharp semantic priors" from high-level vision tasks (like image classification) via knowledge distillation to guide and significantly boost the performance of existing deblurring models.
What Problem Does This Paper Address?
Image deblurring is a classic, ill-posed computer vision problem. When blur is severe—caused by rapid camera shake or fast-moving objects—standard deblurring networks struggle. This is because severe blur causes a massive loss of local detail and introduces semantic ambiguity; it becomes difficult for a network to distinguish object boundaries or even identify what the objects are.
While some specialized methods use semantic segmentation maps to help deblur human faces, this approach fails in general natural scenes. Natural scenes are highly complex, and standard segmentation maps are often too coarse—they provide global category labels but discard the fine textures and local structures essential for high-quality image restoration. The authors aim to bridge this gap by learning a comprehensive prior that captures both global context and sharp local details for general scene deblurring.
Methods
The authors propose a semantic prior-guided framework consisting of two main branches:
- Semantic Prior Learning (SPL) Branch: This branch uses a teacher-student knowledge distillation strategy. A "teacher" network (a high-level classification model pre-trained on sharp images) generates clean semantic features. A "student" network takes the blurry image and learns to mimic these sharp teacher features. To achieve this, the authors use:
- Hierarchical Context Loss (HCL): An enhanced $L_2$ loss that minimizes feature distances across multiple resolutions to maintain local structural scale.
- Cross-Level Connection (CLC) Module: A module that pulls abstract, deep-level semantic features into shallower layers of the student network, enhancing its ability to resolve semantic confusion early on.
- Image Deblurring (IDe) Branch: This is the core restoration network (e.g., a standard UNet or other state-of-the-art architectures). The learned priors are integrated into this branch via a Semantic Prior Embedding (SPE) layer, which consists of:
- Multi-Level Aggregate (MLA) Module: Merges multi-scale priors from different feature dimensions.
- Semantic Attention Transformation (SAT) Module: Dynamically applies scale and shift modulations to the deblurring features, ensuring different image regions receive appropriate weights based on their semantic context and blur severity.
Datasets
The framework was trained and evaluated on three widely recognized datasets:
- GoPro Dataset: A synthetic dataset containing 2,103 training and 1,111 testing pairs exhibiting dynamic motion blur.
- HIDE Dataset: A synthetic dataset specifically tailored for human-aware motion blur, featuring more severe degradation due to frame aggregation.
- RealBlur Dataset (RealBlur-J subset): A real-world dataset consisting of 3,758 training and 980 testing images captured under real-world blurry conditions.
Key Insights and Ideas
- Classification Over Segmentation: Interestingly, features derived from image classification models provide better structural and contextual guidance for general scene deblurring than segmentation maps. Segmentation focuses heavily on edges and shapes, whereas classification features preserve richer internal object contexts and textures.
- Feature-Level vs. Image-Level Priors: Instead of feeding a final discrete map (like a segmentation mask) into the deblurring network, utilizing intermediate, continuous feature representations allows the model to retain a powerful blend of abstract semantics and sharp local textures.
- Privileged Information via Distillation: By utilizing sharp images during training to supervise the student network's interpretation of blurry images, the model effectively learns to "imagine" clean structures even when the input is heavily degraded.
Key Results
The proposed semantic prior (SP) acts as a modular booster that can be plugged into various mainstream architectures, consistently yielding quantitative and qualitative improvements:
- UNet Boost: On the GoPro dataset, adding the semantic prior (
UNet+SP) improved the baseline PSNR by +0.77 dB (from 30.29 dB to 31.06 dB). - State-of-the-Art Enhancements: Integrating the prior into advanced architectures led to significant performance jumps, such as a +0.98 dB improvement for MPRNet and a +0.24 dB improvement for MSSNet.
- Visual Quality: Qualitatively, models equipped with the semantic prior recovered significantly sharper text, cleaner facial features, and fewer structural artifacts in severely blurred zones compared to baseline models.
Limitations and Future Work
Despite its success, the authors note two primary limitations:
- Out-of-Plane Motion Blur: The method struggles with complex 3D or out-of-plane motion (such as hands shaking toward or away from the camera) because rapid depth changes violate the assumptions of the current feature space.
- Tiny Structures: When blurred regions contain exceptionally small details (like tiny text characters), the classification model's categories cannot adequately cover or resolve them, leaving a performance gap between the restored image and the ground truth.
Future work will focus on addressing out-of-plane motion blur and exploring new semantic techniques specifically optimized for tiny object deblurring.
Summary updated