Image super-resolution models are trained on a weighted sum of losses. One kind of loss rewards matching the reference pixels, which yields blurry output. Other kinds reward looking natural: a feature-space distance, or a GAN loss, where a second network tries to tell real images from generated ones. The weights on these terms decide where the model lands on the trade-off between faithfulness and realism, and they are usually either copied from earlier papers or found by retraining the model many times. The metrics used to judge the result are also poorly aligned with people. On GAN-produced distortions in the PIPAL human-judgment dataset, PSNR and SSIM are reported to be anti-correlated with human scores. The popular learned distance LPIPS reaches only 0.4862 correlation, or 0.5135 with anti-aliasing (PIPAL). Newer learned quality models track humans much better, but they can be fooled. Imperceptible perturbations can drop an undefended model's rank correlation on KADID-10k to 0.30–0.57 (BiRQA). If such a model is used directly as a training loss, the generator learns to exploit it.
We propose to use the human-aligned but fragile quality model only to steer a handful of loss weights, and never to push on pixels directly. The weights are adjusted online within a single training run. At each weight update, the change to each weight comes from the inner product between the gradient of that weight's loss term and the gradient of the quality model's score on held-out full-resolution images. Gradient accumulation makes this affordable in two ways. It lets us keep a separate gradient buffer for each loss term. A cached, tile-by-tile backward pass (the "gradient caching" trick from large-batch retrieval) lets us differentiate a whole-image quality score on 2K images without storing the full activation graph. The claim we test is that a three-to-five-dimensional channel of loss weights passes enough signal to move outputs toward human preference, but not enough to game the judge. We also claim this costs one training run plus modest overhead, rather than a Bayesian-optimization campaign of full retrainings.
Core Claim and Mechanism
Let the training loss be $L_{\text{train}}(\theta,\lambda)=\sum_j \lambda_j \ell_j(\theta)$. Here $\theta$ are the model weights, $\ell_j$ are the component losses (pixel L1, a VGG or LPIPS feature loss, and a GAN generator loss), and $\lambda_j\ge 0$ are their weights. Let $Q(\theta)$ be the mean score a full-reference quality model assigns to the model's outputs on a held-out tuning split.
We follow the greedy one-step lookahead of the $T_1$–$T_2$ method (Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters). That method tunes weight decay and dropout by differentiating a validation loss through one optimizer step, at about 30% overhead. For loss weights this derivative needs no Hessian at all. With one update $\theta'=\theta-\alpha P\,\nabla_\theta L_{\text{train}}$, where $\alpha$ is the step size and $P$ is the optimizer's diagonal preconditioner (for Adam, the inverse square-root second moment), the hypergradient is
$$\frac{\partial Q(\theta')}{\partial \lambda_j} = -\alpha\, \nabla_\theta Q(\theta')^{\top} P\, \nabla_\theta \ell_j(\theta).$$
So each weight grows when its loss term's gradient points in the direction that raises the judge's score. The $T_1$–$T_2$ paper needed an identity-Hessian assumption, which it justified with batch normalization. That assumption does not arise here, because the loss weights enter the training gradient linearly.
This formula also resolves a real tension with gradient accumulation. Online hypergradient methods, including the step-size version in Online Learning Rate Adaptation with Hypergradient Descent, assume the weights change at every micro-step. Under accumulation they are frozen for $K$ micro-batches. We apply the derivation at the granularity of the accumulated update. The $\nabla_\theta \ell_j$ are then the per-component sums over the $K$ micro-batches, stored in $J$ separate accumulation buffers instead of one. The lookahead is exact to first order for the update that is actually applied, so no staleness correction is needed. The memory cost is $J$ extra copies of the parameter gradients. That is small for super-resolution networks of a few million to tens of millions of parameters.
Why the Whole Image, and Why Caching
Learned quality models are sensitive to high-frequency content and phase (PIPAL). Structural failures, which feature-based metrics miss because they favor local texture (Xiaoshi Wu), show up only at full scale. Scoring random 128-pixel crops, as training does, would give the judge a biased view. Scoring full 2K outputs and backpropagating to $\theta$ does not fit in memory.
We therefore adapt the three-stage backward pass that Learning With Average Precision and Recall@k Surrogate Loss with Large Batches and Similarity Mixup used for retrieval. In those works the cached quantity is a low-dimensional embedding. In ours it is the output image itself. First, run the super-resolution model tile by tile, without storing the computation graph, to assemble each full output image. Second, evaluate the quality model on the full image and obtain $\partial Q/\partial \hat y$, a gradient image the same size as the output. Third, re-run each tile with enough border padding to cover the network's receptive field, inject the corresponding slice of $\partial Q/\partial \hat y$, and accumulate $\nabla_\theta Q$.
This applies gradient caching to dense spatial outputs, which the file does not report having been done. Here it is a means, not the headline. Its necessity is tested by ablation.
Research Program
We will build the method on top of a standard GAN-based super-resolution training recipe: DIV2K training images at ×4 bicubic degradation, with an L1 + feature + adversarial loss. Fifty of the 800 DIV2K training images are held out as the tuning split, so the DIV2K validation set stays clean for testing. Loss weights are parameterized in log space. They are updated every $N$ logical steps using the tuning-split hypergradient averaged over a few full images. The analysis has three parts.
First, efficiency. We measure the wall-clock overhead over plain training and compare it with the roughly 30% that $T_1$–$T_2$ reports.
Second, gaming. The judge used for tuning and the judges used for evaluation are always different models. We measure the gap between the tuning judge's score and the held-out judges' scores. We run this under both hypergradient tuning and direct use of the same judge as an extra training loss. Our hypothesis is that the weight bottleneck keeps this gap small while the direct loss inflates it.
Third, trajectories. Self-Tuning Networks found that hyperparameter schedules can beat the best static values: on CIFAR-10, validation loss was 0.575 versus 0.636 for Bayesian optimization, and the static values extracted from the schedule did worse when retrained from scratch. We test whether the same holds for perception–distortion weights. We also inspect whether the learned schedule has an interpretable shape, such as pixel loss early and adversarial weight later.
As an extension, we add a PSNR floor to the upper level, handled with a Lagrange multiplier. Several single-run jobs at different floors then trace a perception–distortion frontier. This gives a direct comparison with multi-objective Bayesian optimization, and it respects the fact that no single point on the frontier is optimal for all uses (The Perception-Distortion Tradeoff).
Relation to Prior Work
Three works motivate the problem. The perception–distortion theorem (Blau and Michaeli) says the loss weights are a genuine choice of frontier point, not a nuisance to be averaged away. PIPAL shows that the standard metrics used to make that choice disagree with humans on GAN outputs. BiRQA supplies a better-aligned judge: rank correlation 0.988 on LIVE and 0.979 on CSIQ, 5.5M parameters, and about 15 frames per second at 1080p. Its adversarial fragility result is the constraint that forbids using such judges as direct losses.
The closest existing work is MOBOSR. It tunes super-resolution loss weights with multi-objective Bayesian optimization over quality metrics and reports over 5 dB LR-PSNR improvement over SROOE. It treats each training run as a black box, so every candidate weight setting costs a full training run, and it learns only static weights. We differ by computing weight gradients inside one run and allowing weights to change over time.
LDC-MTL is the closest online method. It is first-order bilevel balancing of multi-task losses with constant per-iteration cost, reaching −1.31% average drop on 40-task CelebA. Its upper objective equalizes loss magnitudes across tasks. For perception versus distortion, equalizing magnitudes is the wrong target, because the terms conflict by theorem. We replace it with an external judge of the output.
The gradient-caching papers supply the memory technique. $T_1$–$T_2$ supplies the greedy hypergradient. We reuse both, but for loss weights, for accumulated updates, and for whole-image judges.
Baselines
The naive reference is fixed literature weights of the form $1.0\cdot L_1 + $ small feature and GAN weights, trained once with the same total steps and accumulation settings. Any method that cannot beat this on held-out judges and human preference has failed.
The strongest existing method is MOBOSR-style multi-objective Bayesian optimization over the loss weights (MOBOSR). It uses the same tuning split and the same tuning judge as its objective, so the comparison isolates the search procedure rather than the target. It will be run under two budgets. The first is equal GPU-hours: our single run, or a small set of runs at different PSNR floors, against as many BO trials as that budget buys. The second is BO's own preferred budget, to see how much compute we save at matched quality. The file gives no reported PSNR, SSIM, or LPIPS table values for MOBOSR on Set5, Urban100, or DIV2K validation; the only reported figure is "over 5dB improvement in LR-PSNR compared to SROOE". Beating it means matching or exceeding its held-out-judge scores and frontier hypervolume at equal GPU-hours, or matching them at a fraction of its hours.
A multi-fidelity baseline, BOHB (BOHB), gets the same GPU-hours, with budget measured in training iterations.
A weight-conditioned single network is trained with loss weights sampled randomly per batch and fed as an input. After training, the weight setting is chosen by the same tuning judge on the tuning split. This is also a one-run method that uses the judge only through the weights. It is the most dangerous competitor, discussed below.
A direct-loss baseline adds the tuning judge as a training loss with a tuned coefficient. It is the gaming control.
Four ablations isolate the proposal's ingredients. The first replaces the human-aligned judge with LPIPS as the upper objective, which tests whether the target matters. The second replaces full-image cached scoring with random-crop scoring, which tests whether caching matters. The third retrains from scratch with the final learned weights held fixed, which tests whether the schedule matters. The fourth drops the preconditioner $P$, which tests whether an Adam-aware inner product matters.
Why not just…?
Why not just run multi-objective Bayesian optimization as MOBOSR does? It works, and it is the baseline to beat. But each trial is a full GAN training run, and it only returns static weights. The Self-Tuning Networks results, 70.30 versus 72.13 validation perplexity on Penn Treebank and 0.575 versus 0.636 validation loss on CIFAR-10 against Bayesian optimization, suggest schedules can matter. That evidence comes from classification and language modeling, not restoration. The equal-GPU-hour comparison settles efficiency. The static-retrain ablation settles whether schedules matter here.
Why not just prune bad settings early with BOHB? Realistic texture and the scores that reward it tend to emerge late in GAN super-resolution training, while early training is dominated by pixel alignment. Low-budget rankings may therefore invert. Downsampled-image proxies (Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks) are worse still, because perceptual judges depend on high-frequency detail. We will measure the rank correlation between BOHB's low-budget and full-budget judge scores. If it is high, BOHB becomes a cheap and serious competitor, and we will say so.
Why not just train one network conditioned on the loss weights and pick weights afterward with the judge? This is a real threat. It is also single-run, it also keeps the judge out of the pixel gradients, and it exposes a whole frontier. Its weaknesses are that it spreads capacity over weight settings nobody will use, and that it cannot express schedules. If it matches our method on held-out judges and human preference at similar cost, our efficiency contribution largely collapses. What would remain is the gaming comparison and the caching technique. That comparison is therefore a decisive experiment, not a side result.
Why not just add the better judge as a training loss? BiRQA's own robustness numbers show that undefended neural judges lose most of their human correlation under optimization pressure. Even its adversarially trained version retains only 0.60–0.84 rank correlation under white-box attack. The direct-loss baseline and the held-out-judge gap test whether that fragility is exploited in practice during training.
Decisive Experiments
All models are evaluated at ×4 on Set5, Set14, BSD100, Urban100, Manga109, and DIV2K validation. We report PSNR and SSIM as distortion measures, LPIPS, the tuning judge's score, and at least one held-out full-reference judge trained on different data. The primary outcome is a pairwise human preference study following the PIPAL pairwise protocol with Elo aggregation. It compares our outputs against the fixed-weight, MOBOSR-style, and weight-conditioned baselines at matched PSNR.
The proposal succeeds if three things hold. Hypergradient tuning must beat fixed weights and at least match Bayesian optimization in human preference at equal or lower GPU-hours, with overhead near or below the 30% reported for $T_1$–$T_2$. The gap between tuning-judge and held-out-judge scores must stay small under hypergradient tuning while it widens under the direct-loss baseline. Removing full-image caching or swapping in LPIPS as the target must measurably reduce human preference.
If the weight-conditioned network ties on all three, we will report that one-run conditioning plus post-hoc selection is sufficient. The surviving claim would then be the narrower one: routing a fragile judge through a few loss weights, whether by gradient or by selection, is a safe way to use it.
References
- PIPAL: A Large-Scale Image Quality Assessment Dataset for Perceptual Image Restoration
- BiRQA: Bidirectional Robust Quality Assessment for Images
- Scalable Gradient-Based Tuning of Continuous Regularization Hyperparameters
- Online Learning Rate Adaptation with Hypergradient Descent
- Human Preference Score: Better Aligning Text-to-image Models with Human Preference
- Learning with Average Precision: Training Image Retrieval with a Listwise Loss
- Recall@k Surrogate Loss with Large Batches and Similarity Mixup
- Self-Tuning Networks: Bilevel Optimization of Hyperparameters using Structured Best-Response Functions
- The Perception-Distortion Tradeoff
- Perceptual-Distortion Balanced Image Super-Resolution is a Multi-Objective Optimization Problem
- LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control
- BOHB: Robust and Efficient Hyperparameter Optimization at Scale
- Speeding up the Hyperparameter Optimization of Deep Convolutional Neural Networks