TL;DR
This paper introduces a novel "All-in-One" image restoration framework that can handle multiple degradation tasks (like denoising, dehazing, deraining) simultaneously with a single model. It achieves this by adaptively selecting the most suitable "expert" modules for each image and task, balancing their contributions to improve overall restoration quality and efficiency.
What Problem Does This Paper Address?
Image restoration, the process of recovering clear images from degraded ones, has seen significant progress with deep learning. However, most methods are specialized for a single degradation type (e.g., only denoising or only dehazing). While "All-in-One" frameworks aim to tackle multiple tasks, they often struggle with:
- Task-specific demands: Different degradations require different restoration strategies (e.g., focusing on local textures vs. global structures).
- Task balancing: Ensuring that improving performance on one task doesn't degrade performance on others.
- Efficiently using task correlations: Leveraging similarities between tasks to improve learning.
- Model adaptability: General frameworks often require separate training for each task or lack flexibility for unseen degradations.
This paper addresses these challenges by proposing a unified framework that intelligently selects and combines specialized restoration modules (experts) based on the input image and the specific task at hand.
Methods
The proposed method, named Multi-Expert Adaptive Selection (MEASNet), consists of two main components:
- Task-Specific Prompts Guide Multi-Expert Selection and Ensemble (STP-G-MESE):
- Task-Specific Prompt Generation (TSPG): This module generates "prompts" that capture information about the image degradation and the specific task. It does this by processing the input image through convolutional layers and then using a Softmax function to create task-relevant prompts. These prompts guide the selection of experts.
- Expert Selection and Ensemble: Based on the generated prompts and the image content features, this part adaptively selects the top-K most suitable "expert" modules from a library. These selected experts are then combined to perform the restoration. The framework also includes a mechanism to balance the selection of experts across different tasks to prevent interference.
- Feature Decomposition and Multi-Expert Ensemble (FD-MEE):
- Feature Decomposition: This component decomposes the image features into low-frequency and high-frequency components. This is motivated by the observation that different degradations affect these components differently (e.g., noise and haze primarily affect high frequencies, while low light affects low frequencies). Separating them allows for more targeted restoration.
- Multi-Expert Ensemble (Global Level): Similar to the pixel-level selection, this part also selects experts, but at a global feature level, considering both the decomposed low- and high-frequency components. This ensures a comprehensive restoration by integrating both local and global image information.
The "experts" themselves are described as multi-layer perceptrons (MLPs) with varying parameters, allowing for diverse restoration capabilities.
Datasets
The paper uses a variety of standard datasets for training and evaluation across different restoration tasks:
- Image Dehazing: SOTS subset from RESIDE dataset.
- Image Deraining: Rain100L dataset.
- Image Denoising: BSD400 and WED datasets for training, BSD68 for testing (with varying noise levels: σ=15, 25, 50).
- Image Deblurring: GoPro dataset.
- Low-Light Image Enhancement: LOL-v1 dataset.
The authors note that they followed the practice of using existing datasets and did not introduce new benchmarks.
Key Insights and Ideas
- Adaptive Expert Selection is Key: The core idea is that no single restoration module is best for all tasks and all images. Dynamically selecting the right "experts" based on image content and task prompts is crucial for effective multi-task restoration.
- Feature Decomposition for Targeted Restoration: Separating image features into low- and high-frequency components allows for more specialized and less interfering restoration processes, leading to better results.
- Task-Content Integration: The framework effectively combines information from both the task (via prompts) and the image content to make informed expert selection decisions.
- Balancing and Collaboration: The multi-expert approach not only caters to individual task needs but also promotes collaboration and knowledge sharing between tasks through shared experts, improving overall efficiency and performance.
Key Results
The proposed method, MEASNet, demonstrates strong performance across multiple image restoration tasks:
- Multi-Task Performance: In a comprehensive evaluation across three degradation tasks (dehazing, deraining, denoising), MEASNet achieved superior results compared to other All-in-One methods. For instance, it improved average PSNR by 0.79 dB over PromptIR and 1.65 dB over AirNet.
- Single-Task Performance: When evaluated on single-task settings, MEASNet also showed competitive or superior results compared to specialized methods, indicating that its All-in-One design does not compromise single-task performance. Visual comparisons in Figure 7 show better restoration quality and fewer artifacts.
- Quantitative Metrics: The paper reports significant improvements in PSNR and SSIM across various tasks and datasets. For example, in deraining, it achieved a PSNR of 39.06 and SSIM of 0.985.
- Efficiency: While using Transformer layers, the authors highlight that their model's parameter size is comparable to or smaller than other Transformer-based methods, and its FLOPs are relatively low.
Limitations and Future Work
- Parameter Size: The authors acknowledge that their model's parameter size is relatively large compared to some CNN-based methods, partly due to the inclusion of Transformer architectures.
- Future Work: The authors plan to explore strategies for further enhancing restoration performance while simultaneously making the models more lightweight.
Summary updated