Which Super-Resolution Training Patches Matter Depends on Budget and on Target Realism

When a model upscales a low-resolution image, many sharp high-resolution images are consistent with the same input. A model trained only to minimize pixel error therefore learns to output the average of those possibilities, which looks blurry. A model trained to look realistic must instead learn to produce one plausible sharp image, and it accepts more pixel error in exchange. This is the perception-distortion tradeoff. Separately, work on data pruning has shown that the best training subset depends on how much data you can keep. With a tiny budget, simple and typical examples work best. With a large budget, hard and unusual examples work best. That second finding has been established almost entirely for classification and language models, while perception-distortion research treats the training set as fixed.

This note proposes that, for image restoration, a patch's training value depends on two quantities: the data budget and the target point on the perception-distortion curve. It also proposes that the standard "difficulty" score mixes two different kinds of hardness that the two ends of the curve value differently. Some patches are hard because the network has not yet learned their structure: edges, line art, repeated geometry. More capacity or more training fixes these. Other patches are hard because the low-resolution input truly does not determine the output: fur, foliage, water ripples. No pixel-error model can do better than the blurry mean on these. A pixel-error model gains little from the second kind. A realism-trained model has to see the second kind, because it is where the model learns texture distributions. If the claim holds, budgeted training sets for super-resolution should be chosen from a two-dimensional difficulty profile, and the right choice should shift predictably with both budget and target realism.

Core Claim

The closest prior work, A Study in Dataset Pruning for Image Super-Resolution, ranks DIV2K patches by the pixel loss of a small pretrained model (FSRCNN). It discards the top 5% as corrupted, a cutoff it calls a "noise ceiling," and trains on the next-hardest 50%, which matches or beats full-data training on PSNR. That study tests only one budget and only pixel-error metrics, and its single loss score cannot tell the two kinds of hardness apart.

The perception-distortion theory of The Perception-Distortion Tradeoff says why the distinction matters. The best possible pixel-error predictor is the conditional mean of the high-resolution image given the low-resolution one. So the leftover error of a strong pixel-error model estimates how ambiguous the patch is, meaning how much its plausible high-resolution versions vary. The error that disappears when moving from a weak model to a strong one estimates the learnable part.

For patch $i$, let $e^{w}_i$ and $e^{s}_i$ be the mean squared errors of a weak and a strong pretrained super-resolution model. Define

$$r_i = e^{w}_i - e^{s}_i, \qquad u_i = e^{s}_i,$$

where $r_i$ is the learnable difficulty and $u_i$ is an estimate of irreducible ambiguity, an upper bound on the posterior variance.

We predict three things.

First, at small budgets, models at every target realism do best on low-$r$, low-$u$ prototype patches. This mirrors the classification findings of SIMS and BOSS.

Second, as the budget grows, pixel-error-trained models gain most from high-$r$ patches.

Third, realism-trained models gain most from high-$u$ patches. The amount of high-$u$ data they need grows with both budget and the weight placed on realism.

The new part is not "prune super-resolution data while measuring LPIPS." It is a testable interaction between budget, target operating point, and a decomposed difficulty score. A selection rule follows directly from that interaction.

Research Program

The work has three parts.

Profiling the patches. We compute $(r_i, u_i)$ for all of roughly 32,000 DIV2K training patches at ×4 bicubic downscaling. The weak model is FSRCNN, matching the pruning study. The strong model is a SwinIR-class network trained with L1 loss on the full set. We check whether $u$ actually measures ambiguity rather than noise in three ways:

  • whether $u$ correlates with the spread of outputs from a stochastic super-resolution sampler on held-out patches;
  • whether it is stable across random seeds of the strong model;
  • whether the top 5% "noise ceiling" patches from the prior study sit in a separate region, such as high $u$ combined with anomalous high-resolution compression statistics, which would let us separate corruption from genuine ambiguity.

Mapping the interaction. Budgets are 5%, 10%, 20%, 50%, and 100% of patches. At each budget we build subsets that vary composition on a grid of $(r, u)$ quantile bands. We train an RRDB generator (the ESRGAN-style backbone) with L1 loss plus a realism term weighted by $\lambda \in \{0, 0.01, 0.1, 1\}$, where the realism term is perceptual feature loss plus adversarial loss. Every run gets the same number of optimizer steps, so smaller subsets are revisited more often. Compute is therefore matched, and only composition varies.

This grid yields, for each budget and $\lambda$, the composition that works best. It also yields the perception-distortion frontier reachable at each budget by varying data alone, by varying $\lambda$ alone, and by varying both.

Building and testing a selection rule. We fit a two-parameter rule from that grid. Patch $i$ is sampled with weight

$$w_i(B,\lambda) \propto \exp\big(a(B)\,\tilde r_i + b(B,\lambda)\,\tilde u_i\big),$$

where $\tilde r_i, \tilde u_i$ are rank-normalized scores, $B$ is the retained fraction, and $a, b$ are smooth functions fit on a subset of the grid. The rule always applies the noise-ceiling trim. We then test it on held-out budgets and held-out $\lambda$ values, and on a second training corpus, OutdoorSceneTraining (OST), to see whether the schedule transfers or must be refit for each dataset.

Baselines

Random subset. The naive reference is a random subset at each budget, trained with the same $\lambda$, steps, and tuning budget. Any selection rule that fails to beat random at matched compute has no value.

Refined descending-loss selection. The strongest existing data method is from A Study in Dataset Pruning for Image Super-Resolution: drop the top 5% by FSRCNN loss, then keep the hardest remaining patches up to the budget. The file does not include its tabulated numbers; for DIV2K pruning PSNR, no reported result found on Lacuna. We therefore rerun it ourselves at every budget and every $\lambda$, using the identical generator and schedule. It is the strongest competitor because it is the only super-resolution pruning method in the file that has been validated, and it already beats full data on PSNR at 50%. Beating it means one of two things at matched budget and $\lambda$:

  • a dominating point, with equal or better PSNR and lower LPIPS on the union of the Set5, Set14, BSD100, Urban100, and Manga109 test sets; or
  • a larger frontier hypervolume in the (PSNR, LPIPS) plane, with gains larger than the seed-to-seed spread.

Loss-weight search on full data. The strongest loss-side method is MOBOSR from Perceptual-Distortion Balanced Image Super-Resolution is a Multi-Objective Optimization Problem. It uses Bayesian optimization over composite loss weights on a fixed DIV2K set. The file reports only a relative figure, more than 5 dB higher LR-PSNR (pixel accuracy measured after re-downscaling the output) than the earlier SROOE method; no absolute per-benchmark numbers found on Lacuna. We run MOBOSR's search on full DIV2K and on our selected subsets, giving both the same total number of trained models. That separates "data shifts the frontier" from "data just saves compute."

Distilled-data reference on OST. For the OST experiment we compare against the synthetic set from GSDD. It reports LPIPS 0.1351 with 700 distilled images versus 0.0903 for the full 10,324-image set. Our 700-image selected subset counts as a win if it falls below 0.1351 LPIPS at comparable SSIM. GSDD trains with its own pipeline, so we report this comparison as indicative, not strictly matched.

Key ablation. The ablation that isolates our central ingredient keeps the budget-dependent schedule but replaces the decomposed $(r, u)$ score with the raw FSRCNN loss. This is a BOSS-style easy-to-hard slide applied to super-resolution, as in BOSS. If it matches the full rule at every $\lambda$, the decomposition contributes nothing, and the work reduces to "transfer budget-aware selection to super-resolution."

Why not just…?

Why not just keep texture-rich patches with an edge filter? The pruning study found that loss-based ranking beats Sobel edge filtering on PSNR A Study in Dataset Pruning for Image Super-Resolution. Realism is a different case, though. Gradient magnitude may track $u$ fairly well, because fur and foliage have strong gradients. We therefore include a gradient-magnitude ranking at every budget and $\lambda$, and report how well it correlates with $u$. If it matches our rule for realism-trained models, the practical conclusion becomes "use gradients for perceptual training, use $r$ for distortion training." That is still a useful result, but a narrower one, and we would report it that way.

Why not just tune the loss weights on the full dataset? This is the most serious threat. It is possible that data composition only moves a model along the same frontier that $\lambda$ already traces, rather than pushing the frontier outward. The matched-model-count comparison against MOBOSR tests this directly. If frontiers built from data and $\lambda$ together have the same hypervolume as $\lambda$-only frontiers on full data, the claim fails except as a compute saving. Two pieces of evidence argue against it. A refined 50% subset already beats full data on PSNR, which suggests full data spends capacity on redundant and corrupted patches A Study in Dataset Pruning for Image Super-Resolution. And in language-model pretraining, data value depends on scale, according to InfoLaw. Neither shows a frontier shift in restoration, which is why the experiment is needed.

Why not just train once on all data and pick the operating point at test time? DiracDiffusion traverses the tradeoff by stopping the reverse diffusion process early. That decides where to sit on the curve. It does not decide what the curve is, and it requires training a full model for each degradation type. Our question is complementary: does curating the training set improve the curve that test-time control then moves along? A secondary experiment trains a small diffusion super-resolution model on random versus selected 20% subsets and compares the early-stopping curves. If both curves coincide, data selection adds nothing for diffusion restorers, and we would state that limit.

Why not just apply hardest-first pruning at every budget? In classification, hardness-only selection collapses at a 10% budget: EL2N reaches 6.37% accuracy versus 33.83% for SIMS on CIFAR-100 SIMS. We do not assume the same collapse happens in super-resolution. The refined descending-loss baseline at 5% and 10% tests it.

Proposed Experiments and Decisive Checks

The main evaluation trains on DIV2K and tests at ×4 on Set5, Set14, BSD100, Urban100, and Manga109. We report:

  • PSNR and SSIM for distortion;
  • LPIPS and NIQE (a no-reference naturalness score) for perceptual quality;
  • a patch-level Fréchet distance to held-out high-resolution patches, as a closer stand-in for the distribution divergence the theory defines.

Every configuration uses three seeds. Because each perceptual proxy is imperfect, every claim must hold on at least two of the three perceptual measures.

Four checks are decisive.

  1. Interaction. In the grid, the best high-$u$ fraction must increase with $\lambda$ at fixed budget. The best high-$r$ fraction must stay flat or decrease with $\lambda$. If the best composition does not depend on $\lambda$, the central claim is false.
  2. Low-budget reversal. At 5% and 10%, prototype-weighted subsets must beat refined descending-loss selection at all $\lambda$. If they do not, the classification-style easy-to-hard transition does not carry over to continuous restoration. That would be a clean negative result.
  3. Decomposition. The full rule must beat the raw-loss schedule ablation, mainly at large $\lambda$, which is where theory says ambiguity matters.
  4. Frontier. At matched model count, frontiers built from data and $\lambda$ together must have larger hypervolume than $\lambda$-only frontiers on full data. The rule must also hold on OST at held-out budgets without being refit.