Telling Misaligned Pairs from Hard Pairs in All-in-One Restoration

Image restoration networks that remove noise, rain, haze and blur with a single model ("all-in-one" restoration) are trained almost entirely on synthetic pairs, where a clean photo is corrupted by a known recipe. The obvious way to close the gap to real cameras is to add real paired data, such as smartphone noisy/clean pairs or real low/high-resolution pairs shot with a zoom lens. Real pairs, however, are dirty supervision. Targets are slightly shifted, exposures and colors drift between shots, and some "clean" targets still contain the degradation. The noisy-label literature has a standard remedy: drop the samples with the largest training loss. That remedy has two known failure modes. It needs the unknown contamination rate to set the drop threshold. It also cannot distinguish a wrong sample from a hard but valid one. In multi-task restoration both failures get worse. Loss scales differ by an order of magnitude across tasks, so a global "drop the top 20%" rule removes dense haze and heavy rain wholesale. And real pairs are harder than synthetic ones precisely because of the domain gap we are trying to learn.

This note proposes a different test for whether a training pair is corrupt, based on the physics of how restoration pairs go wrong. If a pair is bad because of misalignment or photometric drift, most of its loss disappears once the network's output may undergo a tiny shift or a per-image color correction before comparison. If a pair is simply hard, the loss stays. The score is a ratio, so it is comparable across tasks. Its threshold is calibrated on synthetic pairs, where we know which pairs are clean and can inject nuisances deliberately. That turns "guess the contamination rate" into "control the false-flag rate on data we know is clean," a quantity we actually have.

Core Claim

Let $f$ be the restoration network, $(x_i, y_i)$ a degraded input and its target, $L$ the per-sample pixel loss, and $\mathcal{T}$ a small family of nuisance transforms: integer and subpixel shifts up to a few pixels, and a per-channel affine color map. We define the nuisance-explained fraction

$$ s_i = 1 - \frac{\min_{\tau \in \mathcal{T}} L\big(\tau(f(x_i)), y_i\big)}{L\big(f(x_i), y_i\big)}, $$

which is near zero when no cheap transform explains the error and near one when almost all of it is misalignment or color drift. For each degradation family $k$, the threshold $c_k$ is the $(1-\beta)$ quantile of $s$ on that family's known-clean synthetic pairs, with $\beta$ a chosen false-flag rate such as 5%. A pair is down-weighted when $s_i > c_k$.

The claim has two parts. First, this rule keeps hard clean pairs that small-loss selection throws away, including heavy rain, dense haze and high-ISO sensor noise. Second, its performance stays nearly flat when the assumed contamination rate is wrong, because the rule never uses one. That flatness is exactly what loss-rank methods lack.

The score has a blind spot. It only detects corruptions that live in $\mathcal{T}$. A target that still contains rain is not explained by a shift, so for that corruption type we fall back to a loss-rank rule computed within each task. We will measure how much of the benefit survives for that corruption type rather than hide it.

Research Program

The work has three parts.

Instrumented benchmark. We start from the three-task all-in-one setup of PromptIR, which injects learned prompts into a Restormer decoder. Its tasks are denoising on BSD400/WED, deraining on Rain100L and dehazing on RESIDE. We add two real paired sources, SIDD for smartphone noise and DRealSR for camera-specific super-resolution. We then corrupt a controlled fraction $\rho \in \{0, 10, 20, 40\}\%$ of pairs in every source with four corruption types: shifts, color and exposure drift, a mismatched severity target, and residual degradation left in the target. Injected labels give ground truth for measuring how precisely each method flags bad pairs. The untouched real pairs carry whatever natural misalignment they already have.

Estimating the score during training. We compute the score robustly during training. The memorization effect exploited by Co-teaching, in which networks fit clean patterns before they fit noise, implies that $s_i$ is informative only before the network memorizes a shifted target. We therefore compute it from an exponential moving average of the weights and average it over an early window. This borrows the idea from CNLCU of judging a sample by its loss statistics over time rather than at a single step. Weights enter the loss softly, which lets a sample recover later. That is the "forgiveness" property Rockafellian relaxation obtains for classification with a linear-program reweighting that does not carry over to continuous image targets.

Probing the sim-to-real risk. The rule assumes that clean real pairs have roughly the same $s$ distribution as clean synthetic ones. That assumption can fail. Real sensor noise may be partly explained by the color map, for example. We will measure the false-flag rate on a hand-verified clean subset of SIDD and DRealSR and report the calibration shift directly.

This positions the work against three lines of prior research. The noisy-label methods, namely Co-teaching, CNLCU and RRM, motivate the problem and supply the temporal-averaging machinery. All three work on classification labels, and all three need, or partly need, a contamination estimate; the threshold dilemma is documented by Yu et al.. All-in-one restorers, namely AirNet, PromptIR and MEASNet, supply the backbone. All three assume their training pairs are correct. GRIDS shows that task loss scales and gradient interference differ sharply across degradations, which is why a single global threshold is the wrong unit. Finally, the Huber reinterpretation shows that a robust-loss transition should scale with annotation noise. It treats that noise as unstructured Laplace noise. Our point is that restoration noise is structured, as geometric and photometric misregistration, and that this structure can be tested for directly.

Baselines

The naive reference is PromptIR trained on the pooled clean-plus-contaminated data with its standard L1 loss and no selection. Its PSNR at each $\rho$, relative to the same model trained on uncontaminated data, defines the damage that any method has to repair.

The strongest existing restoration systems set the ceiling on clean data. MEASNet reports 39.06 dB PSNR and 0.985 SSIM on Rain100L (MEASNet), against AirNet's 34.90 dB (AirNet). DRCNet reports 40.08 dB on SIDD (DRCNet). For DRealSR, MC-TTDG reports gains of up to 0.57 dB over its baseline but no absolute number usable here. We found no reported result on Lacuna for any sample-selection or noisy-pair method applied to all-in-one restoration, so the robust-training baselines below are ones we implement ourselves.

Loss-rank selection is the most direct competitor, and we run it in two forms. The first is global small-loss dropping in the Co-teaching style, run both with a single network and with the two-network peer exchange. The second is per-task small-loss dropping with CNLCU-style interval lower bounds. We give both the true $\rho$ as their drop rate, which is an oracle advantage. We then sweep the assumed rate from $0.5\rho$ to $2\rho$ to trace their sensitivity curve.

The robust-loss competitor is Charbonnier/Huber training with the transition parameter set per task from the estimated residual scale, following Meyer.

Two ablations isolate what is new in the proposal. The first uses the same soft weighting schedule but replaces $s_i$ with raw loss; this tests the score itself. The second uses $s_i$ with one global threshold instead of per-task thresholds $c_k$; this tests cross-task calibration.

All methods share the same backbone, data, iteration budget and a tuning budget of equal size, eight configurations each, chosen on a held-out synthetic validation split. The minimization over $\mathcal{T}$ adds a fixed extra cost, which we report. Beating the field means two things. At $\rho = 20\%$, the method must close at least half of the PSNR gap between the naive model and the clean-trained model, averaged over the five test sources. And its PSNR must vary by no more than about 0.2 dB across the misspecification sweep, while oracle-tuned loss-rank selection does no worse than it only at the true $\rho$.

Why not just…?

Why not just train with a shift- and color-tolerant loss on every pair? This is the most serious threat. The minimum over $\mathcal{T}$ could simply become the training loss, with no selection at all. The cost is that such a loss also forgives the network's own small misregistrations and color errors on clean pairs, which should blur outputs and cost PSNR on aligned synthetic tests. We run it as a full baseline. If it matches our method on both the real sets and the aligned sets, the selection step adds nothing, and we will report that honestly as the finding. If it wins on the real sets but loses on aligned PSNR, that tradeoff is exactly what selection is meant to avoid.

Why not just pre-register the real pairs offline with optical flow or a global color fit? Offline cleaning handles pure shifts well. It cannot fix parallax or local motion in DRealSR zoom pairs, and it cannot detect targets with a mismatched severity or residual degradation. It is still a real alternative, so we include "register then train naively" as a baseline and break its results down by corruption type. Offline registration and the score can also be combined. If registration alone closes most of the gap, the contribution shrinks to detecting the corruptions that registration cannot see, and the experiments will show that.

Why not just use per-task small-loss selection with a validation-tuned drop rate? Tuning the drop rate on a validation set assumes the validation set shares the training contamination rate and that clean held-out real pairs exist. In practice, when real data are dirty, neither usually holds. Even with oracle $\rho$, loss rank confuses real-but-hard with wrong. Co-teaching and Yu et al. document the under- and over-estimation dilemma in classification. The misspecification sweep and the hard-clean retention measurement in our experiments test whether that dilemma transfers to restoration.

Why not just adapt at test time instead of cleaning the training data? Test-time fine-tuning of the kind MC-TTDG compares against takes 9–19 minutes per patch (MC-TTDG). It also addresses a different problem, a shift at test time, not corrupted supervision at training time. The two approaches are complementary, and we do not claim otherwise.

Proposed Experiments

We run four checks, each of which could decide the question.

Sensitivity. For each method we plot the PSNR averaged over Rain100L, SOTS, BSD68, SIDD and DRealSR against the assumed contamination rate. The prediction is a flat curve for the score-based rule and U-shaped curves for loss-rank methods.

Hard-clean retention. This check exposes why loss rank fails. We measure the fraction of uncorrupted pairs that each method keeps, broken down by quartile of degradation severity: haze density, rain intensity and ISO level. We also compute flag precision and recall against the injected labels, separately for each corruption type. We expect global small-loss selection to lose most of the top haze and rain quartile. We expect the score to keep that quartile while matching loss rank on catching shifted pairs. We expect it to underperform on residual-degradation targets, where the per-task fallback has to carry the load.

Calibration transfer. This check tests the sim-to-real assumption. We compare the false-flag rate on verified clean real pairs with the target $\beta$. A rate far above $\beta$ would mean the synthetic calibration does not transfer, and we would recalibrate $c_k$ on a small real anchor set.

Real-world downstream value. The last check asks whether the cleaner real supervision helps on real scenes at all. We evaluate the resulting all-in-one models on unpaired real hazy images in RTTS, using no-reference quality scores and YOLOv3 detection mAP. As a reference, the dehazing meta-learning method of Towards Generalization on Real Domain for Single Image Dehazing reports a +2.23% mAP gain on RTTS. We note that it is a dehazing specialist rather than an all-in-one model.