Calibrate the Weather, Not the Restorer: Unsupervised Model Selection for Real-World Image Restoration

Networks that remove rain, haze, snow and sensor noise from photos are trained and tuned on synthetic image pairs. Each pair is a clean photo and a copy corrupted by a hand-written formula. When such a network is deployed on a real camera in bad weather, no clean version of the scene exists. The practitioner then has no way to tell which checkpoint, loss weight or adaptation learning rate actually works. This proposal reframes that problem. We cannot score the restored output without ground truth, but we can check whether a corruption simulator produces images that look like the real corrupted inputs, because those inputs are observed. So we tune the parameters of a compound weather-and-sensor simulator until a held-out classifier can no longer tell its outputs from unlabeled real degraded frames. We then use the calibrated simulator to make paired validation data and select the restoration model's hyperparameters on it.

The method moves the "flying blind" problem from the output side, where no signal exists, to the input side, where it does. It borrows a lesson from robotics, where closed-loop simulator tuning beat hand-set randomization. It also yields a built-in warning light: if the classifier still separates real from simulated after calibration, the selection should not be trusted. The claim can be tested directly on a real paired weather benchmark by hiding its clean images and asking whether our selection recovers the configuration that the true pairs would have chosen.

Core Claim

Consider a pool of candidate configurations $h$ for a restoration model: checkpoints, regularization weights, prompt or routing settings, or test-time adaptation step sizes. We claim that ranking this pool by restoration fidelity on pairs from a calibrated simulator agrees with the ranking by fidelity on real pairs much better than three alternatives do. Those alternatives are ranking on the standard synthetic validation split, ranking by no-reference image quality scores, and ranking on a broadly randomized synthetic mixture. We further claim that the residual real-versus-simulated discrepancy after calibration predicts when this agreement breaks down.

Formally, let $\theta$ be the simulator's physical parameters, $P_\theta$ the distribution of simulated degraded images, and $P_{\text{real}}$ the unlabeled real degraded images. Calibration solves

$$\theta^* = \arg\min_\theta \; \mathrm{Acc}\big(P_{\text{real}}, P_\theta\big),$$

where $\mathrm{Acc}$ is the held-out accuracy of a classifier trained to separate the two sets. A value of 0.5 means they are indistinguishable. Selection then picks

$$h^* = \arg\max_h \; \mathbb{E}_{x \sim \text{clean}}\, \mathrm{PSNR}\big(f_h(D_{\theta^*}(x)),\, x\big),$$

where $D_{\theta^*}$ applies the calibrated corruption to a clean image $x$, $f_h$ is the restorer under configuration $h$, and PSNR (peak signal-to-noise ratio) measures pixel fidelity. Here $x$ ranges over clean images that are disjoint from the test scenes.

Where This Sits

The idea combines three pieces of prior work, each in a different role.

The motivation comes from unsupervised model selection. Realistic Evaluation of Test-Time Adaptation Algorithms: Unsupervised Hyperparameter Selection shows that when classification test-time adaptation is tuned with label-free surrogates such as entropy or consistency instead of oracle labels, method rankings invert. EATA, for example, loses to plain TENT under source-accuracy selection. That study covers classification only. It proposes no working unsupervised selector.

The closest method is AutoUAD. It tunes tabular anomaly detectors by measuring discrepancy against synthetic isotropic Gaussian pseudo-anomalies and searching with a Tree-structured Parzen Estimator, a form of Bayesian optimization. It reports OCSVM mean AUC of about 84% versus about 78% for defaults. The authors concede that the Gaussian proxy is too crude for natural images. We keep the idea of a synthetic reference plus Bayesian search. We replace the fixed, maximum-entropy proxy with a structured physical simulator that is itself fitted to observed data. That fitting is what AutoUAD lacks.

The method template comes from sim-to-real robotics. Auto-Tuned Sim-to-Real Transfer trains a classifier to judge whether candidate simulator parameters sit above or below reality from raw video. It reaches 90% real success on Cabinet Slide where domain randomization gets 0%. Traversing the Reality Gap via Simulator Tuning tunes up to 57 physics parameters with differential evolution. It also constrains our design: search diverged as dimensionality grew, so our simulator must stay low-dimensional. Both papers tune dynamics to train policies. We tune image corruption to select models.

Within restoration, the closest work is WeatherCycle. It re-degrades clean images by swapping in the Fourier amplitude spectra of real weather images, which amounts to an implicit, unpaired simulator. However, its contrastive difficulty weights require delicate manual tuning, and it offers no selection protocol. We treat WeatherCycle as one of the models whose hyperparameters need selecting, and its re-degradation idea as a baseline surrogate.

RESIDE found that higher synthetic PSNR can mean worse real-world detection because of over-smoothing. That result constrains us: agreement with real PSNR is necessary but not sufficient, so we also test against downstream detection.

Research Program

We will build a compact, differentiable-free compound degradation simulator with roughly 10 to 15 parameters. Keeping it small follows the divergence result of Collins et al. The simulator has five components:

  • Haze: depth-dependent scattering, with a density parameter and an airlight colour. Depth comes from any off-the-shelf monocular estimate.
  • Rain: a streak layer with density, angle, length and width, plus a global veiling term. This follows the observation in Deep Joint Rain Detection and Removal from a Single Image that heavy rain produces both streaks and mist.
  • Snow: a particle layer.
  • Blur: a motion-blur length.
  • Sensor: photon (Poisson) plus read (Gaussian) noise, followed by a gamma-and-compression stage standing in for the camera's image processing. This follows the non-Gaussian noise evidence in Modeling sRGB Camera Noise with Normalizing Flows.

The order in which the operators are composed is a categorical parameter. Compound effects do not commute, so the order matters.

Calibration uses a few hundred unlabeled real degraded frames from the target condition, together with clean frames from disjoint scenes. The discriminator is a small head on frozen CLIP features, trained fresh at each search step on a split and scored on a held-out split, so that it cannot memorize individual images. The search uses the same TPE procedure as AutoUAD under a fixed budget. We will also try an SPM-style directional classifier as an alternative search driver.

Selection then scores every candidate configuration on calibrated pseudo-pairs. We apply this to three kinds of choices:

  • checkpoints and loss weights of supervised all-in-one restorers trained on synthetic data: PromptIR, AirNet and TransWeather;
  • WeatherCycle's contrastive difficulty weights and cycle weights;
  • learning rate and step count for entropy-based test-time adaptation of PromptIR.

Across these three families the candidate pools differ in ways that matter: early versus late checkpoints, smoothing versus sharpening trade-offs, and adaptation that helps versus adaptation that collapses.

Baselines

The naive reference point is default selection. We take the last checkpoint, or the hyperparameters published by the original authors, and pick the configuration that scores best on the standard synthetic validation split (Rain100L, SOTS, or CDD11 for composites). This is what the field does today.

The strongest existing label-free competitor is no-reference quality selection: choose the configuration minimizing NIQE or BRISQUE on the real degraded frames after restoration. These scores are what DTMIR-Pro reports on real data, including BRISQUE 20.125 on Snow-Realistic. No reported result was found on Lacuna for using such scores as a hyperparameter selector for restoration. We will therefore measure it ourselves.

A second strong competitor is re-degradation consistency in the spirit of WeatherCycle. We restore a real frame, re-apply amplitude-swapped real weather, and score the configuration by how close the result is to the original input. This uses real data without calibrating any physics.

The key ablation is a broadly randomized validation set. It uses the same simulator and the same pseudo-pair PSNR selection, but draws parameters uniformly over wide ranges instead of calibrating them. This isolates whether fitting $\theta$ to real inputs is what matters, as opposed to merely having a richer simulator.

A second ablation uses the calibration objective with FID in place of the held-out classifier. It tests whether the choice of discrepancy measure matters.

The upper bound is selection with a small real paired set: 100 WeatherBench pairs, mirroring the roughly 100 labeled samples that restored near-oracle selection in the adaptation study.

All methods share several controls. They see the same candidate pool and the same unlabeled real frames. The selection methods get equal compute: the TPE budget for calibration counts against our method, and the randomized ablation gets an equal number of simulated pairs. The WeatherBench clean images are touched only for final scoring.

Beating the baselines means two things. First, a higher Spearman rank correlation with real-pair PSNR than every label-free baseline, with bootstrap confidence intervals over scenes that do not overlap. Second, lower regret, defined as the oracle-best PSNR minus the PSNR of the selected configuration. We specifically want the regret gap to the randomized ablation to be significant, not only the gap to defaults.

Why not just…?

Why not just select with NIQE or BRISQUE? RESIDE documents that learned dehazers can win on quality-style criteria through exaggerated sharpening or smoothing while losing on authenticity and downstream detection. DTMIR-Pro notes that its quality-based routing is sensitive to non-weather artifacts. Both findings suggest these scores reward exactly the failure modes a restorer's hyperparameters trade off. The first experiment below measures this directly. If NR-IQA rank correlation on WeatherBench matches ours, the added machinery is not justified, and we will say so.

Why not just randomize the simulator widely and select on that? This is the most serious threat, because it needs no search. The robotics evidence cuts against it: Du et al. report 0% real success for randomization. That evidence comes from control policies, though, not from ranking restorers. The randomized ablation settles the question on our own task. The claim that calibration is the active ingredient fails if the ablation's rank correlation falls within the confidence interval of ours on both WeatherBench and RTTS.

Why not just check consistency by re-degrading the output? Amplitude swapping transfers the global spectrum of real weather. It does not transfer localized streaks, depth-dependent haze or sensor noise. A restorer that returns its input nearly unchanged also scores well on consistency, which is a known degenerate solution of unsupervised surrogates. Including checkpoints that do almost nothing in the candidate pool tests whether this baseline is fooled.

Why not just label a few real pairs? Where that is possible it should be done, and we report it as the upper bound. But WeatherBench needed a purpose-built spraying rig and static scenes to obtain aligned pairs. A deployed camera watching traffic cannot produce them. Our method needs only unlabeled bad-weather frames plus clear-weather frames.

Why not just train on the calibrated simulator instead of only selecting with it? That is a natural follow-up. However, it confounds two questions: whether the simulator matches reality, and whether the network architecture can exploit that match. Selection isolates the first question cheaply, and the answer determines whether training on the calibrated simulator is worth attempting.

Proposed Experiments

Real-pair ranking test on WeatherBench. The primary test runs across its rain, snow and haze conditions, day and night. For each condition we calibrate on unlabeled degraded frames from a set of scenes, then measure rank correlation and regret on held-out scenes against true-pair PSNR and SSIM (structural similarity), for all three model families. As a reference for scale, WeatherBench reports that AdaIR reaches 18.87 dB on real tests when trained on its data versus 18.28 dB when trained synthetically. The whole synthetic-to-real training effect is therefore about 0.6 dB, and selection regret should be judged against that magnitude. No prior unsupervised selection result for restoration on this benchmark was found on Lacuna.

Downstream detection on RESIDE RTTS. RTTS contains real hazy images with bounding boxes. Here we test whether the configuration picked on calibrated haze pairs also gives better detection mAP than the one picked on SOTS. This speaks directly to the gap RESIDE identified between PSNR and downstream usefulness. The RESIDE file text gives no exact mAP figures, so all numbers here will be our own reruns.

Compound mixtures on CDD11. We check that calibration recovers the composition order and parameters when the "real" set is a known synthetic compound. This is a ground-truth recovery test for the simulator, run before any real-data claims.

Warning-light test. Across all conditions, we test whether the discriminator's held-out accuracy after calibration predicts selection regret. For example, night snow may contain sensor effects outside the simulator's family. A positive correlation would let a practitioner know when to fall back to defaults or to collect labels. A flat correlation would mean the method cannot diagnose its own failures, which would be a notable limitation to report.

Whether calibration adds anything. The experiment that decides the proposal is the comparison with the randomized-simulator ablation. If calibration does not beat it, the contribution collapses to "use a better simulator." We would then say that plainly.