Referring Flexible Image Restoration

Runwei Guan, Rongsheng Hu, Zhuhao Zhou, Tianlang Xue, Ka Lok Man, Jeremy S. Smith, Eng Gee Lim, Weiping Ding, Yutao Yue

Abstract

Traditional image restoration models often operate under a fixed paradigm, either targeting a single degradation type or attempting to remove all perceived artifacts indiscriminately. This paper introduces Referring Flexible Image Restoration (RFIR), a novel task that empowers users to restore images containing multiple degradations (e.g., rain, blur, and low light) using specific natural language instructions. We present TransRFIR, a transformer-based architecture that utilizes "Agent" attention modules to achieve linear complexity and state-of-the-art performance. By incorporating a Multiple Degradation Perception (MDP) branch, TransRFIR demonstrates a unique ability to selectively remove artifacts while preserving others according to user intent.

Introduction

Real-world images frequently suffer from multiple simultaneous degradations. A night-time photograph might be obscured by low light, motion blur from camera shake, and rain streaks. While "all-in-one" restoration models have emerged to address multiple issues, they lack flexibility. A user might wish to remove motion blur for clarity but retain atmospheric haze or falling snow for artistic effect. Existing models typically lack the perceptual depth to distinguish between these artifacts or the linguistic grounding to follow nuanced human commands.

The RFIR task addresses this gap by requiring models to not only identify multiple degradations but also to selectively restore the image based on natural language prompts. This necessitates a shift from simple image-to-image mapping to a multi-modal understanding of visual artifacts and textual instructions.

Methodology

The TransRFIR Pipeline

TransRFIR is designed as a multi-modal transformer architecture. It processes a degraded image and a corresponding text prompt through specialized encoders before fusing the information to guide a restoration decoder.

Overview of the TransRFIR pipeline, illustrating the integration of text and image encoders with a restoration-prompt-guided decoder.

Agent Attention (MHASA & MHACA)

A primary challenge in applying Transformers to high-resolution image restoration is the quadratic complexity ($O(N^2)$) of standard self-attention. To overcome this, TransRFIR introduces Agent Attention. This mechanism uses a small set of learnable "Agent" tokens that act as intermediaries. Instead of every pixel attending to every other pixel, they attend to these agents, reducing computational complexity to linear ($O(N)$). This efficiency is applied to both Multi-Head Agent Self-Attention (MHASA) for image features and Multi-Head Agent Cross-Attention (MHACA) for fusing textual prompts with visual data.

Multiple Degradation Perception (MDP)

To ensure the model accurately identifies the artifacts it needs to address, the authors integrated a Multiple Degradation Perception (MDP) branch. This multi-label classification component forces the image encoder to explicitly categorize the degradations present. Experimental data revealed a high correlation (Pearson coefficient > 0.90) between the model's classification accuracy and its eventual restoration quality, confirming that explicit perception is a prerequisite for high-quality flexible restoration.

Experiments and Datasets

To support this new task, the authors developed the RFIR Dataset, a massive synthetic benchmark containing 153,423 samples. It covers five primary degradations—blur, rain, haze, low light, and snow—and includes complex combinations of double and triple degradations. Each sample is paired with text prompts for both global and partial restoration.

To evaluate generalization, TransRFIR was also tested on standard single-degradation benchmarks, including GoPro (blur), Rain100H (rain), LSRW (low light), and SOTS (haze).

Results and Discussion

Performance and Flexibility

TransRFIR achieved state-of-the-art (SOTA) results on the RFIR dataset, outperforming adapted versions of Restormer, NAFNet, and InstructIR in both PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).

Qualitative results demonstrating TransRFIR's ability to selectively restore images based on specific natural language prompts across various degradation types.

As shown in the qualitative results, the model successfully follows negative constraints. For instance, when prompted to "Remove snow," the model can clear snow while leaving other atmospheric effects intact if they were not specified for removal. This level of control is absent in traditional "all-in-one" models.

Efficiency and Generalization

By utilizing Agent Attention, TransRFIR maintains high performance while being significantly faster and less memory-intensive than quadratic attention models. Furthermore, the model demonstrated robust zero-shot generalization, correctly identifying and removing requested degradations even when presented with complex, unseen prompts in triple-degradation scenarios.

Conclusion

TransRFIR represents a significant step toward more interactive and controllable image restoration. By combining linear-complexity attention mechanisms with explicit degradation perception, the model provides a flexible tool for real-world image enhancement. Future work will focus on bridging the gap between synthetic and real-world data and exploring generative approaches, such as Diffusion models, to better reconstruct backgrounds heavily obscured by severe degradations.

Summary updated

BibTeX
@article{guan2025referring,
  author = {Runwei Guan and Rongsheng Hu and Zhuhao Zhou and Tianlang Xue and Ka Lok Man and Jeremy S. Smith and Eng Gee Lim and Weiping Ding and Yutao Yue},
  title = {{Referring Flexible Image Restoration}},
  journal = {Expert Systems with Applications},
  year = {2025},
  doi = {10.1016/j.eswa.2025.126857},
}
Figure