Schedule-Free Iterates as a Perception-Distortion Dial

Image restoration methods such as super-resolution face a known conflict. A model trained to minimize pixel error predicts the average of all plausible sharp images, which looks blurry. A model trained to look realistic invents textures and pays for them in pixel error. Getting a controllable model somewhere between these two ends today usually means training two networks and blending their weights, or training one network per operating point, each under a learning-rate schedule fixed in advance to a set number of steps. This note proposes getting the whole tradeoff curve from a single training run, at any point during that run, by reusing a mechanism that already exists in schedule-free optimizers. These optimizers keep two copies of the weights. One is a fast, noisy "base" sequence $z_t$ that takes constant-size steps. The other is a running average $x_t$ of that sequence, which is the copy normally used at test time.

The central bet is that in a restoration network trained with a realism (adversarial) loss, these two copies land at different places on the perception-distortion plane. The base iterate keeps drifting between hallucinated textures, so it behaves like a sharp but noisy "sample". The average blends many such textures in parameter space. To first order, that is the same as averaging their outputs, which pushes the result toward the low-error, smoother conditional mean. Sweeping a blend coefficient between the two copies would then act as a free test-time dial. Because the base step size never decays, the dial stays open at every step of training instead of collapsing when a learning-rate schedule anneals the two copies together.

Core Claim

Consider a single generator trained with a fidelity loss plus an adversarial loss using Schedule-Free AdamW from The Road Less Scheduled. At any step $t$ we define the family

$$\theta_t(c) = (1-c)\,x_t + c\,z_t, \qquad c \in [0,1],$$

where $x_t$ is the optimizer's averaged sequence, $z_t$ its base sequence, and $c$ a test-time coefficient. The training-time gradient location $y_t$ is one fixed member of this family. We claim three things. First, $\theta_t(c)$ traces a perception-distortion curve that is competitive with ESRGAN-style two-network interpolation. Second, it does so at every stopping time $t$ without retraining. Third, the width of this curve is sustained specifically by the constant base step size.

The quantitative anchor comes from The Perception-Distortion Tradeoff. That paper shows the mean-squared error of a perfectly realistic estimator can be up to twice the minimum achievable error, with the minimum attained by the conditional mean. If base iterates behaved like independent posterior samples, their output average would approach the conditional mean. The gap in mean-squared error between $c=1$ and $c=0$ would then approach this factor of two. Measuring how close real training gets to that ideal, and how far output-averaging and weight-averaging agree, is itself a finding.

This differs from the closest work in a specific way. ESRGAN interpolates between two separately trained networks, each run to a fixed horizon. Multi-scale Recursive and Perception-Distortion Controllable Image Super-Resolution builds the dial into the architecture through a noise-injection weight. The diffusion traversals in Training-Free Rate-Distortion-Perception Traversal With Diffusion, Stage-wise Distortion–Perception Traversal and DiracDiffusion move along the curve at inference, through score scaling, re-noising depth, or early stopping of the sampler. None of these uses the optimizer's own two weight copies as the dial. None addresses what happens to the dial when training stops early or runs longer.

Research Program

The first component is a characterization study, and it addresses an open question. The Road Less Scheduled and Analysis of Schedule Free Non-Convex Optimization cover single-objective minimization only. Last-Iterate Convergence of Regularized Gradient Methods motivates caution, because uncoupled game dynamics can cycle. The research file notes that no anytime guarantees exist for non-convex non-concave generative objectives. We therefore do not claim a theorem. Instead we measure three quantities during training:

  • the output-space spread of the base iterates, meaning how much hallucinated texture changes across nearby $z_t$ snapshots;
  • the gap between $f(x_t)$ and the average output of stored $z$ snapshots, which tests the first-order argument above;
  • whether the raw-to-average distortion gap tracks texture spread.

The second component is the dial itself. We sweep $c$ at test time, including mild extrapolation beyond 1, and compare the resulting curve against competing families.

The third component is stability. We use the fidelity-to-adversarial loss weighting and discriminator settings of ESRGAN. We then compare three discriminator treatments: schedule-free, standard Adam, and a regularized-gradient variant inspired by Last-Iterate Convergence of Regularized Gradient Methods. The goal is to find which combinations keep the base sequence from diverging.

Batch sizes in super-resolution are small. That is the noise-dominated regime where Connections between Schedule-Free Optimizers, AdEMAMix, and Accelerated SGD Variants found schedule-free methods competitive, unlike at 1M-token batches. We will still report a batch-size sweep, because that paper's coupling failure is a direct constraint on the idea. How far away are truly hyperparameter-free learning algorithms? warns that sensitivity moves onto base learning rate and weight decay. Accordingly, every method gets its own tuning of those two quantities under a shared budget.

Baselines

The naive reference point is a single ESRGAN-style generator trained with Adam and cosine decay to a fixed horizon. It yields one point on the plane, and truncating it early shows what schedule brittleness costs.

The strongest direct competitor for a controllable curve is ESRGAN network interpolation from ESRGAN. One network is trained for pixel accuracy, a second is fine-tuned adversarially from it, and their weights are blended. We run it twice: once with its native doubled compute, and once with the total gradient steps matched to our single run. We will count our method as beating it if its perception-distortion curve dominates the matched-compute interpolation curve on Urban100 and DIV2K validation, measured by the area enclosed with respect to a fixed reference point. We will count a tie at half the compute as a practical win. The file contains no reported PSNR or LPIPS numbers for ESRGAN interpolation curves ("no reported result found on Lacuna for ESRGAN network interpolation curves"), so we will reproduce it.

The strongest fixed-point methods set the bar for individual operating points. The first is DECLoss with a relativistic GAN from Uncovering the Over-smoothing Challenge in Image Super-Resolution, which reports 0.080 LPIPS at 28.19 dB PSNR on DIV2K validation at 4x, and 0.093 LPIPS at 24.51 dB on Urban100. The second is EA-Adam from Perception-Distortion Balanced Super-Resolution. It reports more than 0.4 dB PSNR gain and more than 10% LPIPS reduction over Adam on Urban100, at 144 versus 48 training hours. We will report where DECLoss's point falls relative to our curve at the same architecture, using the loss where compatible. For EA-Adam we can only quote relative gains, since the file gives no absolute values.

The key ablation replaces the schedule-free optimizer with Adam plus cosine decay while keeping an exponential moving average of the weights. We apply the identical dial between the moving average and the raw weights, with matched compute and tuning. This isolates whether the constant base step size matters or whether any weight average would do. A second ablation trains the same schedule-free generator with fidelity loss only. The prediction is that the dial then spans almost nothing, which would show that the spread comes from adversarial texture drift and not from optimizer noise alone.

Why not just…?

Why not just interpolate two separately trained networks, as ESRGAN does? This is the default. It works, and we treat it as the bar rather than a straw man. Its costs are two full training runs and two fixed horizons chosen in advance. The matched-compute and truncated-budget comparisons show directly whether one schedule-free run recovers a comparable curve. If it does not, the proposal's practical case reduces to anytime availability alone.

Why not just keep an exponential moving average under a normal schedule and blend it with the raw weights? This is the most serious threat, since it is our mechanism without the schedule-free optimizer. The argument against it is mechanical. As cosine decay drives the learning rate toward zero, the raw weights and their average converge. The dial should therefore be wide mid-training, when the model is under-optimized, and collapse at the end, when the model is good. Schedule-free training keeps a constant base step, so the gap between the two copies stays open. The ablation above measures dial width and curve quality at 25%, 50%, 75% and 100% of budget for both. If the moving-average variant matches at every checkpoint, we will say so and report the finding as "any averaged generator is a dial" rather than a schedule-free result.

Why not just condition the network on a tradeoff knob, like the noise weight in G-MGBP? Multi-scale Recursive and Perception-Distortion Controllable Image Super-Resolution gets continuous control from one compact model. The two approaches are complementary rather than exclusive, because a conditioned model could itself be trained schedule-free. We will compare curves at equal parameter count. Our advantage, if any, is requiring no architectural change and no preference sampling during training.

Why not just use a diffusion model and move along the curve at inference? Score scaling and re-noising need no retraining. However, they cost 23 seconds per image for LMAP-RPS and more than 250 seconds for latent posterior sampling on MS-COCO 4x super-resolution (Stage-wise Distortion–Perception Traversal). A feed-forward model with a weight-blend dial runs a single pass. We will report wall-clock time alongside quality and will not claim quality parity with diffusion samplers.

Proposed Experiments

We train 4x super-resolution generators on DIV2K and evaluate on DIV2K validation, Set5, Set14, BSD100, Urban100 and Manga109. We add RealSR as a check under real camera degradations. Distortion is measured with PSNR and SSIM. Perception is measured with LPIPS, NIQE and the PIRM 2018 perceptual index within its RMSE regions, which lets us check whether any $c$ lands in the challenge regions where ESRGAN and G-MGBP placed.

The first decisive check is anytime behavior. At each checkpoint fraction we compare the area under our curve with three alternatives: a cosine run truncated at that point, a cosine run retrained for exactly that horizon (the oracle), and the moving-average dial. The claim fails if the truncated schedule-free curve is not close to the retrained oracle.

The second decisive check is mechanism. We test whether the mean-squared error ratio between $c=1$ and $c=0$ correlates with measured texture spread across base iterates, and whether it stays below the factor-of-two ceiling. We also test whether $f(x_t)$ agrees with the average output of stored $z$ snapshots. Disagreement would mean weight averaging does something other than output averaging, and the conditional-mean story would be wrong.

The third check is failure analysis on Urban100's repetitive structures, where averaging misaligned textures could produce ghosting rather than clean blur. This is a known weakness that the proposal must report, not hide.