TL;DR
EvoIR is an All-in-One Image Restoration (AiOIR) framework that combines dual-branch frequency feature modulation with an evolutionary loss-scheduling algorithm. By separating low- and high-frequency features into distinct spectral and spatial pathways, while dynamically adjusting loss weights during training, EvoIR achieves state-of-the-art restoration quality across single, mixed, and remote sensing degradation benchmarks with virtually negligible training overhead (~0.1%).
What Problem Does This Paper Address?
Real-world images suffer from diverse degradations—such as noise, rain streaks, haze, blur, and low lighting. Traditional restoration models are task-specific, performing poorly when confronted with unseen or combined corruptions. While recent All-in-One Image Restoration (AiOIR) approaches unify multiple restoration tasks within a single model, two critical challenges remain:
- Spatial-only processing: Most existing architectures operate purely in the spatial domain, ignoring the distinct frequency signatures of different degradations (e.g., rain and noise primarily corrupt high-frequency textures, while low-light and haze alter global, low-frequency illumination).
- Static loss weighting: Training objectives typically balance pixel-level fidelity (e.g., $\ell_1$ loss) and perceptual quality (e.g., MS-SSIM) using fixed, hand-tuned hyperparameters throughout training. This rigid configuration causes gradient conflicts between tasks and fails to adapt to changing optimization dynamics across training stages.
Methods
EvoIR introduces two complementary components within a Restormer-style encoder-decoder backbone:
- Frequency-Modulated Module (FMM):
- Spatial Band-Split: Decomposes feature maps into low-frequency ($X_L$) and high-frequency ($X_H$) branches via a lightweight learnable low-pass convolution.
- Spectral Low-Frequency Gating: Applies a 2D Fast Fourier Transform (FFT) to $X_L$, filters noise-dominated bands using a learned spectral mask, and transforms back via inverse FFT to act as Key/Value representations in subsequent cross-attention.
- Spatial High-Frequency Gating: Processes $X_H$ entirely in the spatial domain using global pooling, lightweight gating, and depthwise convolutions—omitting expensive Fourier transforms while sharply focusing on edges and fine textures.
- Evolutionary Optimization Strategy (EOS):
- Rather than fixing loss weights $\alpha$ (fidelity) and $\beta$ (MS-SSIM perceptual loss), EOS optimizes $(\alpha, \beta)$ on the 2D simplex ($\alpha + \beta = 1$, $\alpha, \beta \ge 0$).
- Every $T = 500$ iterations, network weights are frozen, and a population of candidate weight pairs undergoes elitist selection, convex crossover, and Gaussian mutation evaluated against a held-out validation subset.
- The best-performing candidate is assigned for the next training interval, dynamically tracking the model's moving optimization trajectory.
Input Features ──► [Band Split] ──┬──► Low-Freq ──► FFT ──► Spectral Mask ──► IFFT ──┐
└──► High-Freq ──► Spatial Gating ──► DW-Conv ──────┴──► Fused Output
Datasets
The authors evaluate EvoIR under several standardized benchmarks:
- All-in-One 3D ("N+H+R"): CBSD68 / Kodak24 (Denoising), SOTS-Outdoor / RESIDE-$\beta$ (Dehazing), and Rain100L (Deraining).
- All-in-One 5D ("N+H+R+B+L"): Extends the 3D set with GoPro (Deblurring) and LOL (Low-light enhancement).
- Composite Degradations: The CDD11 dataset, testing 11 combinations of simultaneous degradations (low-light, haze, rain, snow).
- Remote Sensing: MDRS-Landsat, an all-in-one satellite restoration benchmark spanning blur, dark, haze, and noise.
- Single-Task (One-by-One): Independent task-specific evaluations on CBSD68, Kodak24, SOTS, Rain100L, GoPro, and LOL.
Key Insights and Ideas
- Asymmetric Frequency Processing is Efficient: Transforming high frequencies back and forth via FFT/IFFT is computationally heavy and often introduces boundary artifacts. EvoIR limits Fourier transforms strictly to the low-frequency structural stream while keeping high-frequency texture modulation entirely spatial.
- Dynamic Loss Schedules Resolve Gradient Conflicts: AiOIR tasks inherently pull network gradients in conflicting directions. Dynamically updating objective weights via validation fitness prevents the model from over-committing to pixel smoothing early on, accelerating convergence and boosting final PSNR/SSIM.
- Evolutionary Search Can Be Extremely Cheap: Because fitness is evaluated on frozen parameters over a small validation slice for a tiny population ($P=5, G=3$), EOS adds only 1.417 seconds per epoch (0.1% wall-clock overhead) and 84.1 MB peak GPU memory on an 8-GPU setup.
Key Results
- 3D Setting: EvoIR achieves 33.00 dB PSNR / 0.922 SSIM, outperforming state-of-the-art models like AdaIR (+0.31 dB), MoCE-IR (+0.23 dB), and DFPIR (+0.12 dB).
- 5D Setting: Achieves 30.83 dB PSNR, ranking first across all unified baselines (surpassing AdaIR's 30.20 dB and DFPIR's 30.64 dB).
- Composite Degradations (CDD11): Achieves 28.88 dB / 0.885 SSIM, outperforming OneRestore (28.47 dB / 0.878) and winning on 8 out of 11 degradation combinations.
- Remote Sensing (MDRS-Landsat): Without architecture modifications or specialized pre-training, EvoIR (37M parameters) outperforms domain-specialized Ada4DIR-d (41M parameters) across all four degradation types.
Limitations and Future Work
- Deblurring Sub-task Gap: In single-task benchmarks, EvoIR underperforms dedicated deblurring models (e.g., HI-Diff at 33.33 dB vs. EvoIR at 29.57 dB on GoPro). The authors note that blur involves spatially variant point-spread kernels that AiOIR architectures without explicit blur-specific modules struggle to resolve.
- Limited Search Space: EOS is currently constrained to a two-parameter convex combination $(\alpha, \beta)$ balancing fidelity and MS-SSIM. Expanding the search to task-specific loss weights or multi-term objectives (e.g., perceptual VGG, adversarial, or frequency losses) remains an open direction.
Summary updated