Implicit Prompt Learning for Image Denoising

Yao Lu, Bo Jiang, Guangming Lu, Bob Zhang

Abstract

Deep learning models for image denoising often struggle with the trade-off between feature richness and computational overhead. While pre-training on large datasets like ImageNet provides rich features, fine-tuning these models for specific denoising tasks is expensive and prone to overfitting. We introduce IPLID (Implicit Prompt Learning for Image Denoising), a novel framework that leverages frozen, pre-trained backbones through lightweight, learnable "prompts." By inserting ultra-lightweight Linear Prompt (LP) blocks—comprising only 1% of the original parameters—IPLID effectively adapts to complex real-world noise. Our method achieves state-of-the-art performance on benchmarks like SIDD and PolyU while being significantly faster and more parameter-efficient than existing transformer-based models.

Introduction

Modern image denoising requires capturing both fine textures and global context. Traditional approaches either train small models from scratch, which lack feature diversity, or fine-tune massive pre-trained models, which is computationally prohibitive. IPLID addresses this by introducing Implicit Prompt Learning. Instead of explicit text or image prompts, we use internal feature transformations to "nudge" a frozen backbone's knowledge toward the denoising task.

Methodology

The IPLID architecture consists of a frozen encoder (e.g., ResNet-152) and a learnable decoder. The key innovation lies in how the encoder's features are adapted and fused to reconstruct clean images from noisy inputs.

The overall architecture of IPLID, featuring a frozen encoder with Linear Prompt (LP) blocks and a decoder with Compact Feature Fusion (CFF) blocks.

Linear Prompt (LP) Blocks

To adapt the frozen backbone without retraining, we insert LP blocks. These blocks apply a lightweight linear transformation to the features extracted from the backbone. Using learnable coefficients $\alpha, \beta, \text{ and } \lambda$, the LP block adjusts the feature maps to better represent noise-free textures. This mechanism allows the model to leverage the backbone's pre-existing "knowledge" of natural images while specializing in noise removal.

Detailed view of the Linear Prompt (LP) block, which uses learnable parameters to transform features from the frozen backbone.

Compact Feature Fusion (CFF) and Gradient Accumulation

The decoder utilizes Compact Feature Fusion (CFF) blocks, which employ atrous (dilated) convolutions to increase the receptive field without adding significant parameter counts. To ensure effective training of the prompt blocks, a Gradient Accumulation (GA) scheme is used. This scheme applies loss functions at multiple scales throughout the decoder, providing direct supervision to the prompt mechanisms at various depths.

Experimental Results

IPLID was evaluated on standard real-world denoising benchmarks, including SIDD, PolyU, and Nam. Using a frozen ResNet-152 backbone, the model demonstrated exceptional efficiency and performance.

  • Efficiency: The "Tiny" version (IPLID-T) uses only 0.7M parameters, yet it outperforms models like Restormer (26.13M parameters) in speed, running approximately 10x faster (0.014s vs 0.153s per image).
  • Denoising Quality: On the SIDD dataset, IPLID-B achieved a PSNR of 40.05 dB, surpassing MIRNet (39.71 dB) and Uformer (39.77 dB).
  • Detail Preservation: Qualitative analysis shows that IPLID excels at preserving fine details and complex textures that are often lost or blurred by other high-performing architectures.

Conclusion

IPLID represents a significant step forward in parameter-efficient image restoration. By treating pre-trained models as fixed feature extractors and using implicit prompts for adaptation, we achieve state-of-the-art denoising results with a fraction of the computational cost. Future work will explore the application of this prompting strategy to other low-level vision tasks and different backbone architectures.

Summary updated

BibTeX
@inproceedings{lu2024implicit,
  author = {Yao Lu and Bo Jiang and Guangming Lu and Bob Zhang},
  title = {{Implicit Prompt Learning for Image Denoising}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=kHuDsYvsdy},
}
Figure