Phones and other small devices cannot afford to run a large image-restoration network over every pixel. A natural fix is a toolbox: a set of small, specialized restoration networks (for noise, blur, JPEG artifacts, and so on), plus a learned policy that decides, patch by patch and step by step, which tool to apply next and when to stop. Flat sky gets one cheap pass, while a textured, noisy, blurred facade gets a short chain of stronger tools. Two things make this hard to train. The policy is learned by trial and error (reinforcement learning), whose gradient estimates are notoriously noisy. The tools are trained jointly with the policy, which creates a rich-get-richer loop. A tool the policy rarely picks receives little training data, so it stays weak and gets picked even less. It eventually goes dormant, and the toolbox quietly shrinks to a few generalists. That wastes the capacity the edge budget was paying for.
This proposal treats the toolbox as a population, in the sense used by evolutionary optimizers, and treats dormant tools as stagnant individuals. We borrow the stagnation-triggered reset from population metaheuristics. A tool that has stayed unused for a sliding window is replaced by a fresh candidate, cloned and perturbed from a healthy tool of the same compute class. We then add the piece that the reinforcement-learning setting specifically demands: a protected exploration quota. Without it, the policy gradient immediately suppresses the newcomer, whose early advantage is negative. The variance problem is handled separately with a critic-free, per-patch, group-relative baseline. The result is a spatially adaptive restoration system that never runs redundant parallel models at inference. The central question is whether keeping the specialist population alive buys a better quality-versus-latency curve than simpler anti-collapse tricks do.
Core Claim
The claim is that jointly trained, patch-level restoration toolchains suffer from specialist collapse, and that this collapse is what limits their accuracy–latency trade-off, more than gradient variance alone. Two ingredients together fix it. The first is stagnation-triggered replacement of dormant tools with parent-initialized candidates. The second is a protected, off-policy exploration quota that gives each newcomer training data before the policy judges it. With both, the system should beat entropy bonuses, load-balancing losses, frozen pretrained tools, and fixed-schedule resets at matched on-device latency.
The closest existing work comes from three places. Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning (RL-Restore) learns a whole-image policy over 12 small CNNs with a stopping action, and trains tools jointly so they can handle each other's artifacts. It routes at image level, and it does not study whether tools die. Multi-Step Reinforcement Learning for Single Image Super-Resolution gives each 3×3 patch its own actor-critic agent, but chooses among outputs of large models that must all be computed in advance. That precomputation is exactly the inference overhead an edge deployment cannot pay. On the evolutionary side, Evolution-Guided Policy Gradient in Reinforcement Learning (ERL) mixes a population of actors with a gradient learner. Our population is not a population of policies. It is the set of tools the single policy acts over, and injection happens there. That design choice is what makes the idea different from porting ERL to vision.
Research Program
The system. The system has one lightweight policy network and a toolbox of $K$ tools grouped into cost classes. The cheapest class holds look-up-table tools in the style of Toward DNN of LUTs (MuLUT). The middle class holds 3-layer CNNs. The top class holds small blocks built from NAFNet's activation-free SimpleGate design (Simple Baselines for Image Restoration). The image is tiled into overlapping patches. At each of up to $T$ steps, the policy looks at a patch and its step history and chooses a tool or "stop." Only the chosen tool runs. Overlaps are blended to suppress the tile-boundary artifacts noted in the file.
The reward. Each tool's cost is its measured latency on the target device, not its floating-point operation count, because the file notes that routing overhead can erase theoretical savings.
Variance reduction without a critic. We drop the learned value network, since overestimating, non-stationary critics feed their own noise into the policy, as analyzed by Fujimoto et al.. Patches are processed independently apart from blending, so each patch's return is essentially its own. We sample $G$ rollouts per training image and use a leave-one-out baseline computed per patch. This moves the grouped normalization of Group Policy Gradient and GRPO-style methods (Token-Regulated GRPO) from token sequences to spatial agents. It also generalizes the greedy-rollout baseline of Self-Critical Sequence Training.
$$A_{p,g} = R_{p,g} - \frac{1}{G-1}\sum_{g' \neq g} R_{p,g'}, \qquad R_{p,g} = \Delta\mathrm{MSE}_{p,g} - \lambda \sum_{t} c(a_{p,g,t})$$
The symbols are defined as follows. $R_{p,g}$ is the return of patch $p$ in rollout $g$. $\Delta\mathrm{MSE}_{p,g}$ is the reduction in that patch's squared error against ground truth. $c(a)$ is the measured latency of the tool chosen at step $t$. $\lambda$ is a Lagrange multiplier adjusted online so the average latency hits a target budget. Adjusting $\lambda$ this way prevents the two trivial solutions the file warns about, where compute is spent everywhere or nowhere.
Stagnation-triggered injection. Each tool carries a dormancy counter, adapted from the aging operator of Particle Swarm Optimizer with Aging Operator and the dormancy mechanism of Artificial Protozoa Optimizer. The counter increases while the tool's selection share stays below a threshold $\tau$ across a window of $W$ updates. When the counter trips, the tool is replaced. The replacement is initialized as a noise-perturbed copy of the most-used tool in the same cost class, not a random network.
Parent initialization is our answer to the failure mode documented in Genetic Drift Regularization (GDR). In that work, injecting an actor whose parameters had drifted from the population collapsed Walker2D from 2554 (pure evolution strategies) to 249 (unregularized injection). In a toolchain, the analogous drift is a newcomer whose outputs are unlike anything downstream tools have seen. A clone keeps its outputs inside the distribution downstream tools already handle. A small L2 tether to the parent, decayed over the protection period, plays the role of GDR's penalty.
Protection for newcomers. For $M$ updates after injection, a fraction $q$ of patches is forced to the newcomer. These are the patches where the policy's action distribution is least certain among the newcomer's cost class. The forced actions give the new tool supervised gradient. They are excluded from the policy-gradient term, so the forced routing does not bias the policy update. After protection ends, the newcomer competes normally.
The central analysis. The main study measures the number of live tools, the pairwise diversity of tool outputs, and the gradient-norm variance of the policy over training. It then relates each of these to the final quality–latency curve.
Baselines
The naive reference point is a single uniform network run over every pixel at the same device latency. We will use a width-reduced NAFNet, because it is the strongest static efficient design in the file. On SIDD, full-size NAFNet reports 40.30 dB PSNR (Simple Baselines for Image Restoration). The file reports no number for reduced widths, so we will train those variants ourselves with the same data and schedule. Beating this baseline means a higher PSNR at equal measured latency on the target phone chip.
The strongest adaptive competitor is a patch-level extension of RL-Restore, built on the same toolbox. It uses the same joint training and is trained with a standard advantage actor-critic, following Multi-Step Reinforcement Learning for Single Image Super-Resolution. We also run RL-Restore's original whole-image routing. For RL-Restore, the file reports 82.2% fewer computations than VDSR and a 0.2–0.3 dB PSNR advantage on severe unseen distortions. The file reports no absolute DIV2K mixed-distortion PSNR for RL-Restore, so we will reproduce it on our splits. All adaptive methods share the toolbox architecture, the data, the latency table, the number of environment rollouts, and the hyperparameter-search budget, set at 20 configurations each. Beating RL-Restore means a Pareto curve that dominates it across at least three latency budgets, with non-overlapping seed intervals.
We also include the mask-guided scheme of Restore Globally, Refine Locally (MGA). MGA trains an error-predicting mask to send the top-K hard patches to a deeper network. It reports 10–48% fewer FLOPs, and roughly 22% faster RCAN on a Kirin 980. We run it with a single step and two tools, in its native super-resolution setting on Test2K and Test4K, to check whether our machinery even helps when decisions are one-shot.
Three ablations isolate the key ingredients. The first removes injection entirely, leaving the group-relative baseline alone. The second injects without the protected quota. The third injects on a fixed schedule instead of on stagnation, which mirrors the periodic injection of EMQ. A further variant replaces parent initialization with random initialization, to test the drift argument. If the protected-quota ablation matches the full method, our claim about the interaction between injection and the policy gradient is wrong.
Why not just…?
Why not just add an entropy bonus or a load-balancing loss? Both keep tools selected, but they do it by pushing the policy toward uniform use. On an edge budget that means spending expensive tools on patches that do not need them. Neither fixes a tool that is selected but stuck in a poor basin. This is a real threat, because a well-tuned load-balancing loss may prevent most deaths. Both are run as baselines with the same tuning budget. The deciding comparison is PSNR at fixed latency together with the live-tool count. If load balancing reaches the same Pareto curve, injection is unnecessary and we will report that.
Why not just pretrain the tools and freeze them, so they cannot die? Freezing removes collapse by construction. However, RL-Restore found joint training necessary so that tools learn to clean up each other's intermediate artifacts (Crafting a Toolchain). We run a frozen-tool variant. If it matches joint training, collapse is not the bottleneck and the proposal reduces to the variance-reduction piece.
Why not just supervise the router with oracle gains, as MGA does? For one-step, two-way decisions this works: at training time you can run every option and regress on the gain (Restore Globally, Refine Locally). With $K$ tools, $T$ steps, and a stop action, the number of possible chains per patch grows as $K^T$, and the oracle keeps changing as the tools co-train. Our strongest supervised variant labels each step with the best one-step tool. If that greedy oracle matches reinforcement learning at $T \le 3$, the sequential formulation is not worth its cost.
Why not just evolve the router with a pure evolutionary algorithm? ERL reports pure evolution needing more than 20,000 episodes where the hybrid needed about 10,000 on a sparse-reward task (Evolution-Guided Policy Gradient). Evolving thousands of patch decisions per image by whole-image fitness also discards the per-patch reward structure we exploit. We will not run this baseline at full scale. We report a small evolutionary-strategies router at equal rollout count as a sanity check.
Proposed Experiments
The primary benchmark is DIV2K with synthetic mixed blur, noise, and JPEG degradations, following RL-Restore's protocol, including its severe unseen-distortion split (NTIRE 2017). SIDD real smartphone noise (A High-Quality Denoising Dataset for Smartphone Cameras) tests whether the approach helps on a single but spatially varying degradation. Test2K and Test4K test the super-resolution toolbox against MGA and CADyQ. For CADyQ, the file reports 206.5 ms on a 4K image on a Tesla T4 with under 0.05 dB PSNR loss. Metrics are PSNR, SSIM, and measured latency and energy on a Snapdragon-class phone. MuLUT's roughly 240 ms per HD image on a Snapdragon 888 serves as the floor for the cheap tool class.
Four checks decide the outcome. First, over training, the no-injection system must show declining live-tool counts. The full method must hold that count steady, and its final Pareto curve must dominate both the load-balancing and entropy-bonus baselines. Second, removing the protected quota should cause injected tools to die again within about one window $W$. That would show the interaction with the policy gradient is real. Third, random initialization should produce a transient PSNR drop in downstream tools that parent initialization avoids, the vision analog of GDR's collapse. Fourth, the per-patch leave-one-out baseline should cut policy gradient-norm variance relative to the actor-critic baseline, measured over matched training steps. It should also tighten the seed-to-seed spread of final PSNR.
A clean negative result is also informative. If load balancing or frozen tools match the full method, the conclusion is that specialist collapse is not what limits edge toolchains. That would redirect effort toward the tools themselves.
References
- Crafting a Toolchain for Image Restoration by Deep Reinforcement Learning
- Multi-Step Reinforcement Learning for Single Image Super-Resolution
- Evolution-Guided Policy Gradient in Reinforcement Learning
- Toward DNN of LUTs: Learning Efficient Image Restoration With Multiple Look-Up Tables
- Simple Baselines for Image Restoration
- Addressing Function Approximation Error in Actor-Critic Methods
- Group Policy Gradient
- Token-Regulated Group Relative Policy Optimization for Stable Reinforcement Learning in Large Language Models
- Self-Critical Sequence Training for Image Captioning
- Particle Swarm Optimizer with Aging Operator for Multimodal Function Optimization
- Artificial Protozoa Optimizer (APO): A novel bio-inspired metaheuristic algorithm for engineering optimization
- Genetic Drift Regularization: On Preventing Actor Injection from Breaking Evolution Strategies
- Restore Globally, Refine Locally: A Mask-Guided Scheme to Accelerate Super-Resolution Networks
- EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization
- NTIRE 2017 Challenge on Single Image Super-Resolution: Methods and Results
- A High-Quality Denoising Dataset for Smartphone Cameras
- CADyQ: Content-Aware Dynamic Quantization for Image Super-Resolution