Abstract
Traditional image restoration models often operate under a fixed paradigm, either targeting a single degradation type or attempting to remove all perceived artifacts indiscriminately. This paper introduces Referring Flexible Image Restoration (RFIR), a novel task that empowers users to restore images containing multiple degradations (e.g., rain, blur, and low light) using specific natural language instructions. We present TransRFIR, a transformer-based architecture that utilizes "Agent" attention modules to achieve linear complexity and state-of-the-art performance. By incorporating a Multiple Degradation Perception (MDP) branch, TransRFIR demonstrates a unique ability to selectively remove artifacts while preserving others according to user intent.
Introduction
Real-world images frequently suffer from multiple simultaneous degradations. A night-time photograph might be obscured by low light, motion blur from camera shake, and rain streaks. While "all-in-one" restoration models have emerged to address multiple issues, they lack flexibility. A user might wish to remove motion blur for clarity but retain atmospheric haze or falling snow for artistic effect. Existing models typically lack the perceptual depth to distinguish between these artifacts or the linguistic grounding to follow nuanced human commands.
The RFIR task addresses this gap by requiring models to not only identify multiple degradations but also to selectively restore the image based on natural language prompts. This necessitates a shift from simple image-to-image mapping to a multi-modal understanding of visual artifacts and textual instructions.
Methodology
The TransRFIR Pipeline
TransRFIR is designed as a multi-modal transformer architecture. It processes a degraded image and a corresponding text prompt through specialized encoders before fusing the information to guide a restoration decoder.
Agent Attention (MHASA & MHACA)
A primary challenge in applying Transformers to high-resolution image restoration is the quadratic complexity ($O(N^2)$) of standard self-attention. To overcome this, TransRFIR introduces Agent Attention. This mechanism uses a small set of learnable "Agent" tokens that act as intermediaries. Instead of every pixel attending to every other pixel, they attend to these agents, reducing computational complexity to linear ($O(N)$). This efficiency is applied to both Multi-Head Agent Self-Attention (MHASA) for image features and Multi-Head Agent Cross-Attention (MHACA) for fusing textual prompts with visual data.
Multiple Degradation Perception (MDP)
To ensure the model accurately identifies the artifacts it needs to address, the authors integrated a Multiple Degradation Perception (MDP) branch. This multi-label classification component forces the image encoder to explicitly categorize the degradations present. Experimental data revealed a high correlation (Pearson coefficient > 0.90) between the model's classification accuracy and its eventual restoration quality, confirming that explicit perception is a prerequisite for high-quality flexible restoration.
Experiments and Datasets
To support this new task, the authors developed the RFIR Dataset, a massive synthetic benchmark containing 153,423 samples. It covers five primary degradations—blur, rain, haze, low light, and snow—and includes complex combinations of double and triple degradations. Each sample is paired with text prompts for both global and partial restoration.
To evaluate generalization, TransRFIR was also tested on standard single-degradation benchmarks, including GoPro (blur), Rain100H (rain), LSRW (low light), and SOTS (haze).
Results and Discussion
Performance and Flexibility
TransRFIR achieved state-of-the-art (SOTA) results on the RFIR dataset, outperforming adapted versions of Restormer, NAFNet, and InstructIR in both PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity).
As shown in the qualitative results, the model successfully follows negative constraints. For instance, when prompted to "Remove snow," the model can clear snow while leaving other atmospheric effects intact if they were not specified for removal. This level of control is absent in traditional "all-in-one" models.
Efficiency and Generalization
By utilizing Agent Attention, TransRFIR maintains high performance while being significantly faster and less memory-intensive than quadratic attention models. Furthermore, the model demonstrated robust zero-shot generalization, correctly identifying and removing requested degradations even when presented with complex, unseen prompts in triple-degradation scenarios.
Conclusion
TransRFIR represents a significant step toward more interactive and controllable image restoration. By combining linear-complexity attention mechanisms with explicit degradation perception, the model provides a flexible tool for real-world image enhancement. Future work will focus on bridging the gap between synthetic and real-world data and exploring generative approaches, such as Diffusion models, to better reconstruct backgrounds heavily obscured by severe degradations.
Summary updated