EvoIR: Towards All-in-One Image Restoration via Evolutionary Frequency Modulation

Jiaqi Ma, Shengkai Hu, Xu Zhang, Jun Wan, Jiaxing Huang, Lefei Zhang, Salman KhanSubmission to ICLR 2026 · Rejected

TL;DR

EvoIR is an All-in-One Image Restoration (AiOIR) framework that combines dual-branch frequency feature modulation with an evolutionary loss-scheduling algorithm. By separating low- and high-frequency features into distinct spectral and spatial pathways, while dynamically adjusting loss weights during training, EvoIR achieves state-of-the-art restoration quality across single, mixed, and remote sensing degradation benchmarks with virtually negligible training overhead (~0.1%).

What Problem Does This Paper Address?

Real-world images suffer from diverse degradations—such as noise, rain streaks, haze, blur, and low lighting. Traditional restoration models are task-specific, performing poorly when confronted with unseen or combined corruptions. While recent All-in-One Image Restoration (AiOIR) approaches unify multiple restoration tasks within a single model, two critical challenges remain:

  1. Spatial-only processing: Most existing architectures operate purely in the spatial domain, ignoring the distinct frequency signatures of different degradations (e.g., rain and noise primarily corrupt high-frequency textures, while low-light and haze alter global, low-frequency illumination).
  2. Static loss weighting: Training objectives typically balance pixel-level fidelity (e.g., $\ell_1$ loss) and perceptual quality (e.g., MS-SSIM) using fixed, hand-tuned hyperparameters throughout training. This rigid configuration causes gradient conflicts between tasks and fails to adapt to changing optimization dynamics across training stages.

Methods

EvoIR introduces two complementary components within a Restormer-style encoder-decoder backbone:

  • Frequency-Modulated Module (FMM):
  • Spatial Band-Split: Decomposes feature maps into low-frequency ($X_L$) and high-frequency ($X_H$) branches via a lightweight learnable low-pass convolution.
  • Spectral Low-Frequency Gating: Applies a 2D Fast Fourier Transform (FFT) to $X_L$, filters noise-dominated bands using a learned spectral mask, and transforms back via inverse FFT to act as Key/Value representations in subsequent cross-attention.
  • Spatial High-Frequency Gating: Processes $X_H$ entirely in the spatial domain using global pooling, lightweight gating, and depthwise convolutions—omitting expensive Fourier transforms while sharply focusing on edges and fine textures.
  • Evolutionary Optimization Strategy (EOS):
  • Rather than fixing loss weights $\alpha$ (fidelity) and $\beta$ (MS-SSIM perceptual loss), EOS optimizes $(\alpha, \beta)$ on the 2D simplex ($\alpha + \beta = 1$, $\alpha, \beta \ge 0$).
  • Every $T = 500$ iterations, network weights are frozen, and a population of candidate weight pairs undergoes elitist selection, convex crossover, and Gaussian mutation evaluated against a held-out validation subset.
  • The best-performing candidate is assigned for the next training interval, dynamically tracking the model's moving optimization trajectory.
Input Features ──► [Band Split] ──┬──► Low-Freq  ──► FFT ──► Spectral Mask ──► IFFT ──┐
                                  └──► High-Freq ──► Spatial Gating ──► DW-Conv ──────┴──► Fused Output

Datasets

The authors evaluate EvoIR under several standardized benchmarks:

  • All-in-One 3D ("N+H+R"): CBSD68 / Kodak24 (Denoising), SOTS-Outdoor / RESIDE-$\beta$ (Dehazing), and Rain100L (Deraining).
  • All-in-One 5D ("N+H+R+B+L"): Extends the 3D set with GoPro (Deblurring) and LOL (Low-light enhancement).
  • Composite Degradations: The CDD11 dataset, testing 11 combinations of simultaneous degradations (low-light, haze, rain, snow).
  • Remote Sensing: MDRS-Landsat, an all-in-one satellite restoration benchmark spanning blur, dark, haze, and noise.
  • Single-Task (One-by-One): Independent task-specific evaluations on CBSD68, Kodak24, SOTS, Rain100L, GoPro, and LOL.

Key Insights and Ideas

  • Asymmetric Frequency Processing is Efficient: Transforming high frequencies back and forth via FFT/IFFT is computationally heavy and often introduces boundary artifacts. EvoIR limits Fourier transforms strictly to the low-frequency structural stream while keeping high-frequency texture modulation entirely spatial.
  • Dynamic Loss Schedules Resolve Gradient Conflicts: AiOIR tasks inherently pull network gradients in conflicting directions. Dynamically updating objective weights via validation fitness prevents the model from over-committing to pixel smoothing early on, accelerating convergence and boosting final PSNR/SSIM.
  • Evolutionary Search Can Be Extremely Cheap: Because fitness is evaluated on frozen parameters over a small validation slice for a tiny population ($P=5, G=3$), EOS adds only 1.417 seconds per epoch (0.1% wall-clock overhead) and 84.1 MB peak GPU memory on an 8-GPU setup.

Key Results

  • 3D Setting: EvoIR achieves 33.00 dB PSNR / 0.922 SSIM, outperforming state-of-the-art models like AdaIR (+0.31 dB), MoCE-IR (+0.23 dB), and DFPIR (+0.12 dB).
  • 5D Setting: Achieves 30.83 dB PSNR, ranking first across all unified baselines (surpassing AdaIR's 30.20 dB and DFPIR's 30.64 dB).
  • Composite Degradations (CDD11): Achieves 28.88 dB / 0.885 SSIM, outperforming OneRestore (28.47 dB / 0.878) and winning on 8 out of 11 degradation combinations.
  • Remote Sensing (MDRS-Landsat): Without architecture modifications or specialized pre-training, EvoIR (37M parameters) outperforms domain-specialized Ada4DIR-d (41M parameters) across all four degradation types.

Limitations and Future Work

  • Deblurring Sub-task Gap: In single-task benchmarks, EvoIR underperforms dedicated deblurring models (e.g., HI-Diff at 33.33 dB vs. EvoIR at 29.57 dB on GoPro). The authors note that blur involves spatially variant point-spread kernels that AiOIR architectures without explicit blur-specific modules struggle to resolve.
  • Limited Search Space: EOS is currently constrained to a two-parameter convex combination $(\alpha, \beta)$ balancing fidelity and MS-SSIM. Expanding the search to task-specific loss weights or multi-term objectives (e.g., perceptual VGG, adversarial, or frequency losses) remains an open direction.

Summary updated

Abstract

All-in-One Image Restoration (AiOIR) tasks often involve diverse degradation that require robust and versatile strategies. However, most existing approaches typically lack explicit frequency modeling and rely on fixed or heuristic optimization schedules, which limit the generalization across heterogeneous degradation. To address these limitations, we propose EvoIR, an AiOIR‑specific framework that introduces evolutionary frequency modulation for dynamic and adaptive image restoration. Specifically, EvoIR employs the Frequency-Modulated Module (FMM) that decomposes features into high- and low-frequency branches in an explicit manner and adaptively modulates them to enhance both structural fidelity and fine-grained details. Central to EvoIR, an Evolutionary Optimization Strategy (EOS) iteratively adjusts frequency-aware objectives through a population-based evolutionary process, dynamically balancing structural accuracy and perceptual fidelity. Its evolutionary guidance further mitigates gradient conflicts across degradation and accelerates convergence. By synergizing FMM and EOS, EvoIR yields greater improvements than using either component alone, underscoring their complementary roles. Extensive experiments on multiple benchmarks demonstrate that EvoIR outperforms state-of-the-art AiOIR methods.
BibTeX
@misc{ma2025evoir,
  author = {Jiaqi Ma and Shengkai Hu and Xu Zhang and Jun Wan and Jiaxing Huang and Lefei Zhang and Salman Khan},
  title = {{EvoIR: Towards All-in-One Image Restoration via Evolutionary Frequency Modulation}},
  year = {2025},
  eprint = {2512.05104},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.05104},
}
Figure