UniProcessor: A Text-induced Unified Low-level Image Processor

Huiyu Duan, Xiongkuo Min, Sijing Wu, Wei Shen, Guangtao Zhai

Abstract

Traditional low-level vision models are often specialized for single degradation types, and existing all-in-one models struggle with task interference and lack of fine-grained control. We present UniProcessor, a text-driven unified framework capable of handling 30 different image degradation types using a single set of weights. By integrating a vision-language model for perception with a flexible processor backbone, UniProcessor enables precise, multimodal control. This allows users to disentangle and selectively fix specific distortions in complex scenarios. Experimental results demonstrate that UniProcessor achieves state-of-the-art performance across various benchmarks, significantly outperforming existing specialists and all-in-one models.

Introduction

Restoring images affected by blur, noise, or poor lighting typically requires specialized tools. While "all-in-one" systems exist, they face challenges in scaling across diverse corruption types and lack the ability to independently control multiple distortions (e.g., brightening a low-light image while preserving film grain). UniProcessor addresses these hurdles by treating image restoration as a text-induced task, leveraging natural language to guide the restoration process.

Methodology

The UniProcessor architecture consists of three primary components that facilitate a "perceive-then-process" workflow.

  1. Instruction-tuned VQA Module: Based on LLaMA and InstructBLIP, this module acts as the system's "eyes," identifying distortions and generating a "subject prompt" (e.g., "This is a low-quality image with noise distortion").
  2. Context Control Module: This module fuses the subject prompt with a user-defined "manipulation prompt" (e.g., "Remove the noise") using a multimodal Q-Former and CLIP text encoder to produce a Context Control Embedding.
  3. Processor Backbone: A hybrid U-Net-style backbone utilizing ConvFormer blocks for local processing and Transformer blocks for global context. The Context Interaction Modules (CIM) inject text-based control signals into image features via cross-attention.
Overview of the UniProcessor architecture, featuring the VQA module, Context Control module, and Processor Backbone.

Datasets and Evaluation

To train the VQA module, a new Low-level VQA database was created, containing over 70,000 image patches across 30 degradation types paired with natural language Q&A. Restoration training utilized high-quality datasets such as DIV2K and Flickr2K. Evaluation was conducted on standard benchmarks including CBSD68, Urban100, and Kodak24.

Results and Discussion

UniProcessor achieves state-of-the-art performance, outperforming leading models like Restormer and PromptIR. On the CBSD68 dataset, it reached an average PSNR of 30.35 dB. Notably, in snow removal tasks, it showed a 5.5 dB improvement over previous best-in-class methods.

A key advantage is the model's ability to "disentangle" distortions. By adjusting the text prompt, users can selectively remove specific artifacts (e.g., rain streaks) while leaving others (e.g., lighting) untouched.

Visual comparison of UniProcessor results across various single and multiple degradation scenarios.

Conclusion

UniProcessor demonstrates that natural language is a powerful tool for unified image restoration. By combining perception with flexible processing, it provides unprecedented control and superior performance across a wide range of tasks. Future work will focus on reducing computational overhead and improving generalization to complex real-world camera noise.

Summary updated

BibTeX
@inproceedings{duan2024uniprocessor,
  author = {Huiyu Duan and Xiongkuo Min and Sijing Wu and Wei Shen and Guangtao Zhai},
  title = {{UniProcessor: A Text-induced Unified Low-level Image Processor}},
  booktitle = {European Conference on Computer Vision},
  year = {2024},
  doi = {10.1007/978-3-031-72855-6\_11},
}
Figure