Multi-Expert Adaptive Selection: Task-Balancing for All-in-One Image Restoration

Xiaoyan Yu, Shen Zhou, Huafeng Li, Liehuang Zhu

TL;DR

This paper introduces a novel "All-in-One" image restoration framework that can handle multiple degradation tasks (like denoising, dehazing, deraining) simultaneously with a single model. It achieves this by adaptively selecting the most suitable "expert" modules for each image and task, balancing their contributions to improve overall restoration quality and efficiency.

What Problem Does This Paper Address?

Image restoration, the process of recovering clear images from degraded ones, has seen significant progress with deep learning. However, most methods are specialized for a single degradation type (e.g., only denoising or only dehazing). While "All-in-One" frameworks aim to tackle multiple tasks, they often struggle with:

  • Task-specific demands: Different degradations require different restoration strategies (e.g., focusing on local textures vs. global structures).
  • Task balancing: Ensuring that improving performance on one task doesn't degrade performance on others.
  • Efficiently using task correlations: Leveraging similarities between tasks to improve learning.
  • Model adaptability: General frameworks often require separate training for each task or lack flexibility for unseen degradations.

This paper addresses these challenges by proposing a unified framework that intelligently selects and combines specialized restoration modules (experts) based on the input image and the specific task at hand.

Methods

The proposed method, named Multi-Expert Adaptive Selection (MEASNet), consists of two main components:

  1. Task-Specific Prompts Guide Multi-Expert Selection and Ensemble (STP-G-MESE):
    • Task-Specific Prompt Generation (TSPG): This module generates "prompts" that capture information about the image degradation and the specific task. It does this by processing the input image through convolutional layers and then using a Softmax function to create task-relevant prompts. These prompts guide the selection of experts.
  2. Expert Selection and Ensemble: Based on the generated prompts and the image content features, this part adaptively selects the top-K most suitable "expert" modules from a library. These selected experts are then combined to perform the restoration. The framework also includes a mechanism to balance the selection of experts across different tasks to prevent interference.
  3. Feature Decomposition and Multi-Expert Ensemble (FD-MEE):
    • Feature Decomposition: This component decomposes the image features into low-frequency and high-frequency components. This is motivated by the observation that different degradations affect these components differently (e.g., noise and haze primarily affect high frequencies, while low light affects low frequencies). Separating them allows for more targeted restoration.
  4. Multi-Expert Ensemble (Global Level): Similar to the pixel-level selection, this part also selects experts, but at a global feature level, considering both the decomposed low- and high-frequency components. This ensures a comprehensive restoration by integrating both local and global image information.

The "experts" themselves are described as multi-layer perceptrons (MLPs) with varying parameters, allowing for diverse restoration capabilities.

Datasets

The paper uses a variety of standard datasets for training and evaluation across different restoration tasks:

  • Image Dehazing: SOTS subset from RESIDE dataset.
  • Image Deraining: Rain100L dataset.
  • Image Denoising: BSD400 and WED datasets for training, BSD68 for testing (with varying noise levels: σ=15, 25, 50).
  • Image Deblurring: GoPro dataset.
  • Low-Light Image Enhancement: LOL-v1 dataset.

The authors note that they followed the practice of using existing datasets and did not introduce new benchmarks.

Key Insights and Ideas

  • Adaptive Expert Selection is Key: The core idea is that no single restoration module is best for all tasks and all images. Dynamically selecting the right "experts" based on image content and task prompts is crucial for effective multi-task restoration.
  • Feature Decomposition for Targeted Restoration: Separating image features into low- and high-frequency components allows for more specialized and less interfering restoration processes, leading to better results.
  • Task-Content Integration: The framework effectively combines information from both the task (via prompts) and the image content to make informed expert selection decisions.
  • Balancing and Collaboration: The multi-expert approach not only caters to individual task needs but also promotes collaboration and knowledge sharing between tasks through shared experts, improving overall efficiency and performance.

Key Results

The proposed method, MEASNet, demonstrates strong performance across multiple image restoration tasks:

  • Multi-Task Performance: In a comprehensive evaluation across three degradation tasks (dehazing, deraining, denoising), MEASNet achieved superior results compared to other All-in-One methods. For instance, it improved average PSNR by 0.79 dB over PromptIR and 1.65 dB over AirNet.
  • Single-Task Performance: When evaluated on single-task settings, MEASNet also showed competitive or superior results compared to specialized methods, indicating that its All-in-One design does not compromise single-task performance. Visual comparisons in Figure 7 show better restoration quality and fewer artifacts.
  • Quantitative Metrics: The paper reports significant improvements in PSNR and SSIM across various tasks and datasets. For example, in deraining, it achieved a PSNR of 39.06 and SSIM of 0.985.
  • Efficiency: While using Transformer layers, the authors highlight that their model's parameter size is comparable to or smaller than other Transformer-based methods, and its FLOPs are relatively low.

Limitations and Future Work

  • Parameter Size: The authors acknowledge that their model's parameter size is relatively large compared to some CNN-based methods, partly due to the inclusion of Transformer architectures.
  • Future Work: The authors plan to explore strategies for further enhancing restoration performance while simultaneously making the models more lightweight.

Summary updated

Abstract

The use of a single image restoration framework to achieve multi-task image restoration has garnered significant attention from researchers. However, several practical challenges remain, including meeting the specific and simultaneous demands of different tasks, balancing relationships between tasks, and effectively utilizing task correlations in model design. To address these challenges, this paper explores a multi-expert adaptive selection mechanism. We begin by designing a feature representation method that accounts for both the pixel channel level and the global level, encompassing low-frequency and high-frequency components of the image. Based on this method, we construct a multi-expert selection and ensemble scheme. This scheme adaptively selects the most suitable expert from the expert library according to the content of the input image and the prompts of the current task. It not only meets the individualized needs of different tasks but also achieves balance and optimization across tasks. By sharing experts, our design promotes interconnections between different tasks, thereby enhancing overall performance and resource utilization. Additionally, the multi-expert mechanism effectively eliminates irrelevant experts, reducing interference from them and further improving the effectiveness and accuracy of image restoration. Experimental results demonstrate that our proposed method is both effective and superior to existing approaches, highlighting its potential for practical applications in multi-task image restoration.
BibTeX
@article{yu2025multi,
  author = {Xiaoyan Yu and Shen Zhou and Huafeng Li and Liehuang Zhu},
  title = {{Multi-Expert Adaptive Selection: Task-Balancing for All-in-One Image Restoration}},
  journal = {IEEE TCSVT},
  year = {2025},
  doi = {10.1109/tcsvt.2024.3516074},
}
Figure