TL;DR
TransRFIR introduces a new task called Referring Flexible Image Restoration (RFIR), where a model restores images containing multiple degradations (like rain, blur, and low light) based on specific natural language instructions. By utilizing novel "Agent" attention modules, the model achieves linear complexity and state-of-the-art performance, allowing users to selectively remove certain artifacts while intentionally leaving others.
What Problem Does This Paper Address?
Traditional image restoration follows two main paradigms: task-specific models (e.g., just deblurring) or "all-in-one" models that attempt to remove every degradation they perceive. However, real-world images often suffer from multiple simultaneous issues—imagine a rainy night shot that is both dark, blurry, and streaked with rain.
The authors identify a critical gap: existing models lack "flexibility." A user might want to remove the motion blur from a camera shake but keep the atmospheric haze or falling snow for aesthetic reasons. Current models struggle to perceive multiple degradations within a single image and cannot follow nuanced human commands to selectively restore only parts of the image.
Methods
The authors propose TransRFIR, a transformer-based multi-task architecture. Its core innovations lie in efficiency and perception:
- Agent Attention (MHASA & MHACA): Standard self-attention has quadratic complexity ($O(N^2)$), which is computationally expensive for high-resolution images. TransRFIR introduces "Agent" tokens—a small set of learnable parameters that act as intermediaries. Instead of every pixel attending to every other pixel, they attend to these agents, reducing complexity to linear ($O(N)$). This is applied to both self-attention (image features) and cross-attention (fusing text prompts with image features).
- Multiple Degradation Perception (MDP): To ensure the model actually "understands" what is wrong with the image, the authors added a multi-label classification branch. This forces the encoder to explicitly identify the types of degradations present, which the authors prove significantly boosts the quality of the final restoration.
- Multi-Modal Fusion: The system uses the ALBERT language model to process text prompts (e.g., "Remove rain and blur"), which then guide the image decoder through the staged restoration process.
Datasets
The authors introduced a massive new benchmark:
- RFIR Dataset: A synthetic dataset containing 153,423 samples. It covers five basic degradations: blur, rain, haze, low light, and snow. Crucially, it includes combinations of double and triple degradations and provides corresponding text prompts for both partial and global removal.
- Generalization Benchmarks: To prove the model's versatility, they also evaluated it on standard single-degradation datasets including GoPro (blur), Rain100H (rain), LSRW (low light), and SOTS (haze).
Key Insights and Ideas
- The "Aha" Moment for Perception: The authors found a high correlation (Pearson coefficient > 0.90) between the model's ability to classify degradations and its ability to restore them. This confirms that explicit "perception" is a prerequisite for high-quality flexible restoration.
- Selective Restoration: Unlike "all-in-one" models that are "tempted" to remove everything, TransRFIR demonstrates it can follow negative constraints (e.g., removing blur while leaving haze) by aligning textual instructions with specific feature maps in the decoder.
- Efficiency without Sacrifice: By using Agent Attention, the model achieves a better tradeoff between accuracy and computational cost (FLOPs) compared to vanilla Transformers or CNN-based architectures like NAFNet.
Key Results
- SOTA Performance: TransRFIR achieved the highest PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) scores on the RFIR dataset compared to adapted versions of Restormer, NAFNet, and InstructIR.
- Linear Scaling: The model maintains high performance while being significantly faster and less memory-intensive than quadratic attention models, especially as the "agent size" (the number of summary tokens) increases.
- Zero-Shot Generalization: Even when tested on "unseen" prompts not found in the training set, the model correctly identified and removed the requested degradations in complex triple-degradation scenarios.
Limitations and Future Work
- The "Rain-Snow" Confusion: The authors noted that when rain and snow co-exist, the model sometimes struggles to distinguish between them perfectly due to visual similarities and pixel-wise occlusions.
- Synthetic vs. Real-World: While the RFIR dataset is large, it is synthetic. The authors acknowledge the need to collect more real-world samples to bridge the domain gap.
- Generative Potential: The current model is discriminative. The authors suggest that integrating generative approaches (like Diffusion models) might better handle cases where degradations heavily obscure the background.
Summary updated