Referring flexible image restoration

Runwei Guan, Rongsheng Hu, Zhuhao Zhou, Tianlang Xue, Ka Lok Man, Jeremy S. Smith, Eng Gee Lim, Weiping Ding, Yutao Yue

TL;DR

TransRFIR introduces a new task called Referring Flexible Image Restoration (RFIR), where a model restores images containing multiple degradations (like rain, blur, and low light) based on specific natural language instructions. By utilizing novel "Agent" attention modules, the model achieves linear complexity and state-of-the-art performance, allowing users to selectively remove certain artifacts while intentionally leaving others.

What Problem Does This Paper Address?

Traditional image restoration follows two main paradigms: task-specific models (e.g., just deblurring) or "all-in-one" models that attempt to remove every degradation they perceive. However, real-world images often suffer from multiple simultaneous issues—imagine a rainy night shot that is both dark, blurry, and streaked with rain.

The authors identify a critical gap: existing models lack "flexibility." A user might want to remove the motion blur from a camera shake but keep the atmospheric haze or falling snow for aesthetic reasons. Current models struggle to perceive multiple degradations within a single image and cannot follow nuanced human commands to selectively restore only parts of the image.

Methods

The authors propose TransRFIR, a transformer-based multi-task architecture. Its core innovations lie in efficiency and perception:

  • Agent Attention (MHASA & MHACA): Standard self-attention has quadratic complexity ($O(N^2)$), which is computationally expensive for high-resolution images. TransRFIR introduces "Agent" tokens—a small set of learnable parameters that act as intermediaries. Instead of every pixel attending to every other pixel, they attend to these agents, reducing complexity to linear ($O(N)$). This is applied to both self-attention (image features) and cross-attention (fusing text prompts with image features).
  • Multiple Degradation Perception (MDP): To ensure the model actually "understands" what is wrong with the image, the authors added a multi-label classification branch. This forces the encoder to explicitly identify the types of degradations present, which the authors prove significantly boosts the quality of the final restoration.
  • Multi-Modal Fusion: The system uses the ALBERT language model to process text prompts (e.g., "Remove rain and blur"), which then guide the image decoder through the staged restoration process.

Datasets

The authors introduced a massive new benchmark:

  • RFIR Dataset: A synthetic dataset containing 153,423 samples. It covers five basic degradations: blur, rain, haze, low light, and snow. Crucially, it includes combinations of double and triple degradations and provides corresponding text prompts for both partial and global removal.
  • Generalization Benchmarks: To prove the model's versatility, they also evaluated it on standard single-degradation datasets including GoPro (blur), Rain100H (rain), LSRW (low light), and SOTS (haze).

Key Insights and Ideas

  • The "Aha" Moment for Perception: The authors found a high correlation (Pearson coefficient > 0.90) between the model's ability to classify degradations and its ability to restore them. This confirms that explicit "perception" is a prerequisite for high-quality flexible restoration.
  • Selective Restoration: Unlike "all-in-one" models that are "tempted" to remove everything, TransRFIR demonstrates it can follow negative constraints (e.g., removing blur while leaving haze) by aligning textual instructions with specific feature maps in the decoder.
  • Efficiency without Sacrifice: By using Agent Attention, the model achieves a better tradeoff between accuracy and computational cost (FLOPs) compared to vanilla Transformers or CNN-based architectures like NAFNet.

Key Results

  • SOTA Performance: TransRFIR achieved the highest PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) scores on the RFIR dataset compared to adapted versions of Restormer, NAFNet, and InstructIR.
  • Linear Scaling: The model maintains high performance while being significantly faster and less memory-intensive than quadratic attention models, especially as the "agent size" (the number of summary tokens) increases.
  • Zero-Shot Generalization: Even when tested on "unseen" prompts not found in the training set, the model correctly identified and removed the requested degradations in complex triple-degradation scenarios.

Limitations and Future Work

  • The "Rain-Snow" Confusion: The authors noted that when rain and snow co-exist, the model sometimes struggles to distinguish between them perfectly due to visual similarities and pixel-wise occlusions.
  • Synthetic vs. Real-World: While the RFIR dataset is large, it is synthetic. The authors acknowledge the need to collect more real-world samples to bridge the domain gap.
  • Generative Potential: The current model is discriminative. The authors suggest that integrating generative approaches (like Diffusion models) might better handle cases where degradations heavily obscure the background.

Summary updated

Abstract

Highlights•A novel task: Referring Flexible Image Restoration, prompt-based degrade-removal.•A novel dataset called RFIR, containing 153,423 samples with five degradation types.•A multi-task model for specific degradation-removal and multi-deg classification.•A highly efficient attention module called MHASA with linear complexity.•A novel fusion method called MHACA with better performance than vanilla MHCA.
BibTeX
@article{guan2025referring,
  author = {Runwei Guan and Rongsheng Hu and Zhuhao Zhou and Tianlang Xue and Ka Lok Man and Jeremy S. Smith and Eng Gee Lim and Weiping Ding and Yutao Yue},
  title = {{Referring flexible image restoration}},
  journal = {Expert Systems with Applications},
  year = {2025},
  doi = {10.1016/j.eswa.2025.126857},
}
Figure