Abstract
Traditional low-level vision models are often specialized for single degradation types, and existing all-in-one models struggle with task interference and lack of fine-grained control. We present UniProcessor, a text-driven unified framework capable of handling 30 different image degradation types using a single set of weights. By integrating a vision-language model for perception with a flexible processor backbone, UniProcessor enables precise, multimodal control. This allows users to disentangle and selectively fix specific distortions in complex scenarios. Experimental results demonstrate that UniProcessor achieves state-of-the-art performance across various benchmarks, significantly outperforming existing specialists and all-in-one models.
Introduction
Restoring images affected by blur, noise, or poor lighting typically requires specialized tools. While "all-in-one" systems exist, they face challenges in scaling across diverse corruption types and lack the ability to independently control multiple distortions (e.g., brightening a low-light image while preserving film grain). UniProcessor addresses these hurdles by treating image restoration as a text-induced task, leveraging natural language to guide the restoration process.
Methodology
The UniProcessor architecture consists of three primary components that facilitate a "perceive-then-process" workflow.
- Instruction-tuned VQA Module: Based on LLaMA and InstructBLIP, this module acts as the system's "eyes," identifying distortions and generating a "subject prompt" (e.g., "This is a low-quality image with noise distortion").
- Context Control Module: This module fuses the subject prompt with a user-defined "manipulation prompt" (e.g., "Remove the noise") using a multimodal Q-Former and CLIP text encoder to produce a Context Control Embedding.
- Processor Backbone: A hybrid U-Net-style backbone utilizing ConvFormer blocks for local processing and Transformer blocks for global context. The Context Interaction Modules (CIM) inject text-based control signals into image features via cross-attention.
Datasets and Evaluation
To train the VQA module, a new Low-level VQA database was created, containing over 70,000 image patches across 30 degradation types paired with natural language Q&A. Restoration training utilized high-quality datasets such as DIV2K and Flickr2K. Evaluation was conducted on standard benchmarks including CBSD68, Urban100, and Kodak24.
Results and Discussion
UniProcessor achieves state-of-the-art performance, outperforming leading models like Restormer and PromptIR. On the CBSD68 dataset, it reached an average PSNR of 30.35 dB. Notably, in snow removal tasks, it showed a 5.5 dB improvement over previous best-in-class methods.
A key advantage is the model's ability to "disentangle" distortions. By adjusting the text prompt, users can selectively remove specific artifacts (e.g., rain streaks) while leaving others (e.g., lighting) untouched.
Conclusion
UniProcessor demonstrates that natural language is a powerful tool for unified image restoration. By combining perception with flexible processing, it provides unprecedented control and superior performance across a wide range of tasks. Future work will focus on reducing computational overhead and improving generalization to complex real-world camera noise.
Summary updated