SAM-DEBLUR: Let Segment Anything Boost Image Deblurring

Siwei Li, Mingxuan Liu, Yating Zhang, Shu Chen, Haoxiang Li, Zifei Dou, Hong ChenICASSP 2024 · DOI

TL;DR

This paper introduces SAM-Deblur, a novel framework that leverages the powerful segmentation capabilities of the Segment Anything Model (SAM) to significantly improve image deblurring performance, especially on out-of-distribution datasets. By integrating SAM's structural priors, SAM-Deblur addresses the "mode collapse" issue and enhances model generalization.

What Problem Does This Paper Address?

Image deblurring aims to remove blur artifacts from images, a crucial task for many downstream applications like object detection and autonomous driving. However, a major challenge is handling non-uniform blurring, which makes the problem ill-posed and limits the generalization ability of existing deep learning models. These models, while performing well on in-distribution data (like the GoPro dataset), often struggle when applied to real-world datasets with different blur characteristics, leading to phenomena like "mode collapse" where the model produces anomalous or repetitive outputs. This severely restricts their practical utility.

Methods

SAM-Deblur tackles this problem by integrating semantic priors from the Segment Anything Model (SAM) into the deblurring process. The framework operates in three main stages:

  1. Preprocessing with SAM and Mask Dropout: Blurred images are first processed by SAM to obtain segment masks. To enhance model robustness during training, a "mask dropout" technique is employed, where some of the generated masks are randomly discarded. This prevents the model from becoming overly reliant on specific regions.
  2. Mask Average Pooling (MAP) Unit: A novel plug-and-play component called the Mask Average Pooling (MAP) Unit is introduced. This unit takes the blurred image and SAM-generated masks as input. It first encodes the image features and then, for each mask, averages the encoded features within the masked region. This aggregated information, representing structural priors, is then reassigned to the corresponding pixels.
  3. Feature Fusion and Deblurring: The features generated by the MAP Unit are concatenated with the original blurred image features. This combined input is then fed into a standard deblurring network (in this case, NAFNet) to produce the final sharp image. The MAP unit is designed to be co-trained with the deblurring network.

Datasets

  • Training: GoPro dataset [1] (2,103 training images, 1,111 test images).
  • Evaluation (Out-of-Distribution):
    • RealBlurJ [14] (980 images)
    • REDS [2] (3,000 images)
    • ReLoBlur [15] (395 images)

Key Insights and Ideas

  • SAM's Robustness to Blurred Images: The paper highlights that SAM, due to its extensive pretraining on a massive dataset (SA-1B), can accurately segment objects even in blurred images, making it a valuable source of structural priors for deblurring.
  • Bridging the Gap to OOD Generalization: By incorporating semantic information from SAM, SAM-Deblur effectively regularizes the solution space, leading to improved generalization on datasets with different blur characteristics than the training set.
  • Mitigating "Mode Collapse": The proposed MAP unit and mask dropout strategy are shown to effectively address the "mode collapse" issue observed in standard deblurring models when applied to OOD data.
  • Plug-and-Play Integration: The MAP Unit is designed as a flexible component that can be seamlessly integrated into existing deblurring networks, making the approach broadly applicable.

Key Results

  • Improved Performance on OOD Datasets: SAM-Deblur significantly outperforms baseline methods on RealBlurJ, REDS, and ReLoBlur. For instance, it achieves PSNR improvements of 0.05, 0.96, and 7.03 respectively, compared to a model trained without SAM priors. SSIM also sees notable increases.
  • Reduced Mode Collapse Rate (MCR): The proposed method drastically reduces the MCR on OOD datasets, with reductions of 0.20%, 2.6%, and 40.76% on RealBlurJ, REDS, and ReLoBlur, respectively.
  • Minor Impact on In-Distribution Data: On the GoPro dataset (in-distribution), the performance gains are minimal, indicating that SAM-Deblur primarily enhances generalization rather than significantly boosting performance on data similar to the training set.
  • Qualitative Improvements: Visual results (Fig. 4) demonstrate that SAM-Deblur produces sharper and more artifact-free deblurred images on OOD datasets compared to other methods.

Limitations and Future Work

  • Computational Cost: While NAFNet itself is streamlined, the addition of SAM for segmentation and the MAP unit will introduce some computational overhead, especially during the preprocessing stage. The paper doesn't explicitly quantify this overhead.
  • Dependence on SAM Quality: The performance of SAM-Deblur is inherently tied to the quality of masks generated by SAM. While SAM is robust, extreme blur or unusual object types might still pose challenges.
  • Future Work: The authors suggest that their framework opens new avenues for practical applications of deblurring algorithms in real-world scenarios, implying potential for further exploration and refinement of SAM-based priors for various image restoration tasks.

Summary updated

BibTeX
@inproceedings{li2024sam,
  author = {Siwei Li and Mingxuan Liu and Yating Zhang and Shu Chen and Haoxiang Li and Zifei Dou and Hong Chen},
  title = {{SAM-DEBLUR: Let Segment Anything Boost Image Deblurring}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2024},
  doi = {10.1109/icassp48485.2024.10445844},
}
Figure