Spend Refinement Only Where the Restorer Failed

Real photographs are often damaged by several things at once. A night street scene may be dark, hazy and rained-on in the same pixels. A newer family of "all-in-one" restoration networks tries to undo such compound damage in a single forward pass. The best of them, AllRestorer, reports 33.72 dB PSNR and 0.9436 SSIM on the composite-degradation benchmark CDD-11. Whatever these networks get wrong is passed straight to the output, and nothing gets a second look.

The usual fix is a second pass. That can be a small add-on network bolted onto the output, or a loop that drafts and then revises. Both options waste effort. An add-on processes every pixel exactly once. A loop processes every pixel every round, and its cost grows linearly with the number of rounds. Yet the leftover errors after compound restoration are probably concentrated in a few places, such as dark-and-hazy sky, rain over fine texture, or noise in shadows.

This note proposes a small, reusable correction network that sits after any frozen restorer. It first estimates where the current image is still wrong. It then corrects only those patches, and repeats until its own error estimate says further passes will not help. The compute spent on each image therefore follows how badly that image was restored rather than a fixed schedule.

Core Claim

A single small network with two heads can make iterative refinement both targeted and self-terminating without a separate verifier. The first head is cheap and predicts a per-pixel map of the remaining error. The second head is heavier and computes a correction, and it is applied only to image tiles that the first head flags. Attached to a frozen compound-degradation restorer, this loop should give a better quality-versus-latency curve than three alternatives: running the same corrector once over the whole image, running it a fixed number of times over the whole image, or spending the same compute on fine-tuning or enlarging the restorer.

The selection-and-stopping rule is simple. Let $y_t$ be the image after round $t$. Let $\hat e_t(p)$ be the predicted absolute error at pixel $p$, and let $\bar e_t(k)$ be its average over tile $k$. Round $t+1$ corrects the set of tiles

$$ S_t=\{k:\bar e_t(k)>\tau\}, $$

where $\tau$ is a single threshold. The loop stops when $S_t$ is empty or when a round budget $T_{\max}$ is reached. Sweeping $\tau$ traces out the whole compute-quality curve from one trained model.

What is new is the combination of three properties, none of which prior work has together. The corrector is a plug-in that works with any restorer. It runs repeatedly. It chooses its own regions and stopping point from its own error prediction, and it is tested on overlapping degradations.

Relation to Prior Work

The plug-in idea comes from Boosting Image Restoration via Priors from Pre-Trained Models. That paper adds a module of fewer than 1M parameters, guided by CLIP and Stable Diffusion features, after Restormer or SNR-Net. It reports 40.22 dB on SIDD and a 4.02 dB gain on LOL-real. It runs once, over the whole image, on single degradations. We keep the architecture-agnostic interface but make the module repeat, act selectively, and target composite damage.

The motivation for selective correction comes from two directions. Navab et al. reclassify only the uncertain boundary pixels of a segmentation network, which shows that a decoupled refiner can usefully focus on a subset of the image. Iterative Partial Refinement re-generates randomly chosen regions of a diffusion sample. It names informed region selection as an open problem, because random selection hurts when the resampled fraction is large. Its compute also reaches about 13× a single pass at 50 iterations. Our error head is the informed selector that work lacks, and the stopping rule addresses its linear cost.

Two prior works shape the training design.

  • Training on its own intermediate outputs. RL-Restore chains small CNN tools on mixed blur, noise and JPEG artifacts, with a learned stop action. It reports 82.2% fewer computations. It also shows that each tool must be trained on the partly restored images produced by earlier tools, its "middle state" problem. We adopt that lesson: our corrector is trained on its own unrolled outputs. Unlike RL-Restore, we use one weight-shared module rather than a toolbox, select spatial regions rather than tools, and need no reinforcement learning, because the error head is supervised directly by the ground truth.
  • Overfitting to one backbone's artifacts. The modular-refinement literature warns that add-on modules can overfit to one backbone's artifact patterns. So we train on outputs from several restorers and test on a held-out one.

The compound-degradation networks, OneRestore, AllRestorer and M2Restore, supply the frozen first stage. They all decide in one pass how strongly to restore each region. If the error head shows that their residual errors really are spatially concentrated and predictable, that supports the decoupled view held by MPRNet and RL-Restore over the purely joint view. MPRNet's three stages run unconditionally on every patch, and we treat that as the fixed-compute reference.

Finally, Meta-Reflection shows, for language reasoning, that a multi-pass reflection loop can be compressed into one pass with comparable quality, cutting latency from 951 ms to 246 ms. That result is a direct threat to the value of iteration, and we test it explicitly below.

Research Program

The corrector. The network takes the degraded input and the current estimate and outputs two things. The first is a low-resolution error map from a shallow head. The second is a correction residual from a deeper CNN-plus-windowed-attention head, which runs only on gathered tiles of 64×64 pixels with overlap. Corrected tiles are blended back with feathered edges. We target about 1–2M parameters. Optionally the network can use frozen CLIP features, as the plug-in paper does. This is an ablation rather than a core ingredient, because that paper's own literature warns that foundation-model priors can hallucinate detail.

Training. The restorer stays frozen and is used to produce first-stage outputs on CDD-11 training images. We unroll the corrector for up to $T_{\max}=4$ rounds. Tile selection during training is sampled, mixing high-error and random tiles, so the corrector sees a range of intermediate states. The loss has two parts. The correction head is trained with an L1 loss against ground truth at every round. The error head is trained to regress the true absolute error of the current estimate. Gradients are stopped between rounds to keep memory bounded.

To test transfer across restorers, we train on outputs of OneRestore and a Restormer-style all-in-one model, then test attached to AllRestorer, and rotate which restorer is held out.

Analysis. We will measure three things. First, how well the predicted error map ranks the true error. Second, how compute allocation varies across CDD-11's single, double and triple degradation categories. Third, where the corrector refuses to act.

Baselines

Do nothing. The naive reference is the frozen restorer alone. For the strongest one, AllRestorer, the reported CDD-11 result is 33.72 dB PSNR and 0.9436 SSIM, from the AllRestorer paper. We will re-run it on our hardware to measure latency, because no AllRestorer latency figure was found on Lacuna. Every gain is measured relative to this re-run, and also against OneRestore's reported 28.72 dB for context.

Single-pass full-image corrector. This is our module run once over every pixel, with the error head unused. It is the faithful CDD-11 analogue of the plug-in approach in Boosting Image Restoration via Priors from Pre-Trained Models. It is the most dangerous competitor, because it captures most of the benefit of "a second look" at minimal complexity. We also build a version made deeper so that its FLOPs equal the average FLOPs of our adaptive loop. To beat it, our method must reach equal or higher PSNR at lower mean latency, or at least 0.3 dB higher at matched mean latency, across three training seeds.

Fixed-round, full-image iteration. This applies the same corrector $K$ times everywhere, for $K$ from 1 to 4, and plays the role of an unconditional multi-stage design like MPRNet.

Iterative Partial Refinement-style random tiles. This uses the same trained corrector, but tiles are chosen uniformly at random with the same per-round tile count as our method. It is the ablation that isolates the error head.

Oracle selection. This chooses tiles using the true error and gives the upper bound on what selection can achieve.

Spend the budget on the restorer. The last baseline fine-tunes AllRestorer's final decoder stages on CDD-11, and separately widens them, with the added FLOPs matched to our module's average cost.

All baselines share the same training images, the same number of optimizer steps or a matched GPU-hour budget, and the same learning-rate search over five values. Wall-clock time is measured on one GPU at batch size 1 with full-resolution images, and tile gather and scatter time is counted.

Why not just…?

Why not just run a plug-in refiner once over the whole image? This is the strongest alternative, and it may well suffice. If the leftover errors after compound restoration are spread fairly evenly, selection buys nothing. A single dense pass also avoids the per-round overhead that RL-Restore's sequential design was criticized for. The comparison against the FLOPs-matched single-pass corrector settles this directly. If the single pass matches us at equal latency, we would report that iteration is unnecessary and keep only the error-map analysis.

Why not just fine-tune or enlarge the all-in-one restorer? The plug-in paper argues that monolithic models hit diminishing returns as they grow. OneRestore's 5.98M model also beat the larger Restormer on CDD-11. But those comparisons were not compute-matched to a refiner, so they are suggestive rather than decisive. Our matched fine-tune and widen baselines answer the question for compound degradation. Even if they tie on quality, they cannot give per-image adaptive cost and they are not reusable across restorers. Those two properties are what we would then claim.

Why not just cascade single-task specialists? The compound-degradation literature and RL-Restore's middle-state analysis show that a stage fixing one artifact tends to amplify another, and that specialists not trained on such intermediate images fail. We include a dehaze → derain → low-light-enhancement cascade of AllRestorer-family components as a sanity reference only, since it lacks the joint training on intermediate states that RL-Restore found necessary.

Why not just distill the loop into one pass, as Meta-Reflection does for language? This is a real threat. We will train a single-pass student to match the four-round output. If the student matches the adaptive loop's quality at lower latency, the honest conclusion is that targeted iteration is mainly a training-time teacher for the corrector. That would still be a useful finding, and we would report it as such.

Proposed Experiments

Main experiment. We report PSNR, SSIM and LPIPS against mean wall-clock latency on CDD-11, broken down by the number of degradation components. For each method we report three things: quality at matched latency, latency at matched quality, and the full curve traced by sweeping $\tau$.

Secondary benchmarks. We use the All-weather Outdoor-Rain split, where rain and haze co-occur. There M2Restore reports 31.65 dB average PSNR, 0.936 SSIM and 0.17 s on 224×224 images on an RTX 3090. We also use DIV2K with mixed blur, noise and JPEG artifacts against RL-Restore. For DIV2K, no absolute PSNR figure from RL-Restore was found on Lacuna, so we re-run it.

Decisive checks. The project rests on four checks.

  • The predicted error map must rank true tile error well, with Spearman correlation above roughly 0.6 on held-out restorers. If it cannot, selection cannot work.
  • Error-guided tiles must beat random tiles at equal compute. This is the core ingredient.
  • Average rounds and corrected area should increase from single to triple degradations. Otherwise the compute is not following difficulty.
  • The corrector must transfer to a restorer it never saw in training, losing less than half its gain. This tests the interface-overfitting concern raised in the modular-refinement literature.

Failing the first two checks would falsify the core claim, not merely weaken it.