UP-Restorer: When Unrolling Meets Prompts for Unified Image Restoration

Minghao Liu, Wenhan Yang, Jinyi Luo, Jiaying LiuAAAI 2025 · OpenReview

TL;DR

This paper introduces "Up-Restorer," a novel all-in-one image restoration method that unifies various degradation types (rain, haze, noise) by simulating an iterative optimization process using diffusion models guided by dynamic prompts. This approach effectively learns degradation-specific priors, leading to significant performance improvements across diverse restoration tasks.

What Problem Does This Paper Address?

Image restoration (IR) aims to recover high-quality images from degraded versions. While deep learning has excelled at single-task restoration (e.g., denoising, deblurring), real-world scenarios often involve complex, combined degradations. Existing "all-in-one" methods struggle with adaptability and scalability due to reliance on specific degradation modeling or multi-stage training. The challenge is to create a single, unified model that can effectively handle a wide range of degradations without explicit task-specific training or complex architectures.

Methods

The core innovation of Up-Restorer lies in its prompt-based unrolling network that simulates an iterative optimization process derived from the Maximum A Posteriori (MAP) principle. This is achieved by:

  • Unrolling MAP Optimization: The paper frames image restoration as an optimization problem and "unrolls" its solution, breaking it down into simpler, iterative sub-problems. This is inspired by methods like the Alternating Direction Method of Multipliers (ADMM).
  • Diffusion Model Integration: A score-based diffusion model is used as the backbone for the restoration process.
  • Dynamic Prompt Generation: The key novelty is the introduction of "Prompt Blocks" that dynamically generate prompts. These prompts are used to estimate and bypass physics-related degradation variables (like transmission maps, atmospheric light, and noise maps) at each iteration. This allows the model to adapt to the specific characteristics of the input image.
  • Unified Degradation Model: A single model is proposed that can handle multiple degradations by formulating them within a unified mathematical framework.
  • Learnable Queries in Diffusion Transformer: To further enhance adaptability, learnable query parameters are integrated into the Diffusion Transformer's self-attention mechanism, allowing it to dynamically adjust to varying degradations.
  • Two-Stage Training: The model is trained in two stages: first, to minimize noise prediction error (L1 loss), and second, to optimize the restored image using a combination of SSIM and L1 losses.

Datasets

The paper evaluates Up-Restorer on several standard datasets:

  • Denoising: BSD400 (training), BSD68 (testing), WED (training).
  • Deraining: Rain100L (training and testing).
  • Dehazing: Outdoor Training Set (OTS) for training and Synthetic Objective Testing Set (SOTS) for testing from the RESIDE dataset.
  • Additional tasks (for broader evaluation): GoPro (deblurring), Snow100K (desnowing), RealSR (super-resolution).

Key Insights and Ideas

  • Unrolling as a Path to Unified Restoration: Simulating iterative optimization processes (like ADMM) within a diffusion framework provides a principled way to build unified restoration models that can learn complex degradation priors.
  • Prompts as Dynamic Conditioners: Dynamically generated prompts are crucial for enabling a single model to adapt to diverse and unknown degradation types. They act as learned "instructions" for the diffusion process.
  • Bridging Physics and Data-Driven Approaches: By unrolling physics-based optimization and integrating it with data-driven diffusion models, Up-Restorer benefits from both the interpretability of physical models and the power of deep learning.
  • Efficiency through Iterative Refinement: Breaking down the restoration into smaller, prompt-guided steps within the diffusion framework leads to more stable and effective learning of degradation characteristics.

Key Results

  • State-of-the-Art Performance: Up-Restorer achieves the best overall performance when averaged across multiple degradation tasks in the "All-in-One" setting, outperforming previous methods like PromptIR.
  • Significant Gains: The paper reports a significant gain of 0.10 dB over the previous best all-in-one method (PromptIR) in the "All-in-One" setting.
  • Strong Single-Task Performance: Up-Restorer also demonstrates competitive results in single-task settings, particularly excelling in rain removal.
  • Perceptual Quality: The method shows superior perceptual quality metrics (LPIPS, HFD-IQA, Hyper-IQA) compared to other state-of-the-art methods.
  • Lightweight Design: The prompt-related modules add only a small percentage (around 8-10%) to the total parameter count, keeping the model relatively efficient.

Limitations and Future Work

  • Computational Cost: While the model is designed to be efficient, diffusion models inherently involve iterative processes, which can still be computationally intensive.
  • Generalization to Extreme Cases: The paper focuses on common degradations. Future work could explore its performance on more extreme or novel combinations of degradations.
  • Further Exploration of Prompt Components: The paper suggests that further research into the design and interaction of prompt components could lead to even more robust and adaptable restoration capabilities.
  • Extension to Other Vision Tasks: The prompt-based unrolling framework could potentially be extended to other image-to-image translation tasks beyond restoration.

Summary updated

Abstract

All-in-one restoration needs to implicitly distinguish between different degradation conditions and apply specific prior constraints accordingly. To fulfill this goal, our work makes the first effort to create an all-in-one restoration via unrolling from the typical maximum a-posterior optimization function. This unrolling framework naturally leads to the construction of progressively solving models, which are equivalent to a diffusion enhancer taking as input dynamically generated prompts. Under a score-based diffusion model, the prompts are integrated for propogating and updating several context-related variables, i.e. transmission map, atmospheric light map and noise or rain map progressively. Such learned prompt generation process, which simulates the nonlinear operations in the unrolled solution, is combined with linear operations owning clear physics implications to make the diffusion models well reguarlized and more effective in learning degradation-related visual priors. Experimental results demonstrate that our method achieves significant performance improvements across various image restoration tasks, realizing true all-in-one image restoration.
BibTeX
@inproceedings{liu2025up,
  author = {Minghao Liu and Wenhan Yang and Jinyi Luo and Jiaying Liu},
  title = {{UP-Restorer: When Unrolling Meets Prompts for Unified Image Restoration}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  url = {https://openreview.net/forum?id=ksX4iP9pnm},
}
Figure