Where Should Robustness Live in a High-Resolution Diffusion Model?

Diffusion models are generators that learn to turn random noise into images by repeatedly removing a little noise. They are trained on web-scale image collections that contain a steady fraction of damaged images: compression blocks, watermarks, pasted text, and stitched patches. The usual answer to bad training targets in classification is a robust loss, meaning one whose penalty is bounded, so a few wildly wrong examples cannot pull the model too far. That answer comes with a known cost. Bounded losses underfit, because their gradients go flat exactly when the model is uncertain. High-resolution diffusion training already struggles in the same spot. At large image sizes, the noisiest training steps are the ones that teach global layout, and these steps are under-trained to begin with.

This note argues that the two problems collide in a predictable place. If a robust loss is applied uniformly, it will starve global structure, and the damage will grow with resolution. We propose to treat denoising at each noise level and each frequency band as a separate task in a multi-task problem. Robustness is applied only to the bands and noise levels where corrupted images actually look like outliers. A cheap gradient-magnitude balancer keeps the bounded tasks from going silent. The deliverable is a measured map of where robustness helps and where it hurts, a training rule derived from that map, and a test of whether the rule's advantage over uniform robustness grows from 128 to 512 pixels.

Core Claim

Our central claim is that underfitting from robust losses in diffusion is not uniform. Its location is set by how uncertain the model is at each noise level and frequency band.

At high noise, even a perfectly trained denoiser has large errors on clean images, because many clean images are consistent with a very noisy input. A bounded loss therefore saturates on almost every sample there, clean or not. This is the flat-gradient-under-uncertainty failure documented for classification by Normalized Loss Functions for Deep Learning with Noisy Labels, Zhou et al. and Ishiguro et al.. simple diffusion showed that raising resolution increases the signal-to-noise ratio at every step. Its remedy, shifting the noise schedule toward higher noise, puts more of training into exactly this saturated regime. A uniformly robust loss should therefore hurt global layout more as resolution grows.

At low noise and in fine-detail bands, the picture is reversed. Clean residuals are small, while compression blocks, watermarks and pasted patches leave large, localized residuals. Robustness can separate good targets from bad ones only in these bands, and it costs little there.

The new element is a loss that routes robustness by noise level and frequency band, measured from the model's own clean-error statistics, together with a balancer that adjusts gradient magnitudes across bands. To our knowledge, no prior work in the file trains a diffusion model with a bounded or symmetric loss at all. Nor does any prior work examine how multi-task gradient balancing interacts with losses whose gradients are bounded by design.

Relation to Prior Work

Three robust-loss papers motivate and supply the method. Robust Loss Functions under Label Noise for Deep Neural Networks proved that losses with a constant sum over classes tolerate noise. Generalized Cross Entropy introduced a tunable bridge between cross entropy and mean absolute error. The Active-Passive Loss work (Ma et al.) combined a term that pushes up the correct class with a term that pushes down the others, which counters underfitting. All three study classification. All three apply one robustness level everywhere and re-tune it per dataset; Active-Passive Loss, for example, moves from weights of 1 and 1 to 10 and 0.1 on CIFAR-100. We keep their bounded-loss idea but make the robustness level vary by band and noise level.

Ye et al. constrains the design. It shows that equal-magnitude gradients fail to prioritize hard but clean samples. The routing rule must therefore leave high-noise, coarse bands under ordinary squared error, where the hard clean examples live.

simple diffusion supplies the setting and the representation. It contributes the shifted schedule, the wavelet transform that splits an image into frequency channels without losing information, and multiscale losses. It uses plain squared error and assumes clean data, which is the assumption we drop.

ResDiff is the closest structural analogue. It splits super-resolution into a low-frequency network and a diffusion model for high-frequency residuals, trained with pixel, Fourier and wavelet losses. It assumes clean paired targets, trains its stages sequentially, and never asks which band should be robust.

RotoGrad supplies the balancing idea. It separates task conflict into gradient-size imbalance and gradient-direction conflict. We borrow only its size-balancing half, for two reasons. Bands are summed rather than routed through task-specific layers, and the authors themselves note that full rotations become expensive on large layers. RotoGrad also assumes ordinary losses. Our question is whether size balancing can restore learning when a task's gradient is small by design. If the balancer simply rescales a bounded loss back toward squared error, it would cancel the robustness, and we test for that explicitly.

Selective Underfitting in Diffusion Models frames the analysis. It argues that diffusion models fit tightly only in thin regions around training images and extrapolate elsewhere. We use its lens to ask whether bounded losses shrink those regions, which would blur samples, or merely exclude corrupted images from them.

Research Program

Write $x$ for a training image after wavelet decomposition into bands $b$, and $\hat{x}_b(t)$ for the model's prediction of band $b$ at noise level $t$. Write $e_b(t)=\lVert \hat{x}_b(t)-x_b\rVert^2$ for the per-sample, per-band error. The training objective is

$$\mathcal{L}=\sum_{b}\sum_{t} w_b(t)\,\rho_{b,t}\big(e_b(t)\big),$$

where $\rho_{b,t}$ is either the identity (ordinary squared error) or a bounded transform $\rho(e)=\tau\,(1-e^{-e/\tau})$, and $w_b(t)$ is a weight. The bounded transform is the regression analogue of the cross-entropy-to-mean-absolute-error bridge.

The routing decision for each band and noise level is based on a separability score. To compute it, we first train briefly on a small trusted clean subset and record the error distribution on clean images at each $(b,t)$. We then compare that distribution with the error distribution on a held-out set carrying injected corruptions. The bounded transform is switched on only where the two distributions are well separated, and $\tau$ is set at a high quantile of the clean error. Where they are not separated, squared error is kept.

The weights $w_b(t)$ are then adjusted online so that each band's gradient norm on the shared network tracks a common target, as in RotoGrad's size balancing. The balancer acts only through $w_b(t)$, not inside $\rho$. It can therefore raise a quiet band's overall signal but cannot remove the per-sample cap on corrupted images.

The work proceeds in three steps. The first is diagnostic. We apply a uniform bounded loss to a pixel-space diffusion model of the U-ViT type at 128, 256 and 512 pixels and measure per-band gradient norms and held-out clean error. This tests whether coarse bands at high noise are the ones that saturate, and whether that saturation grows with resolution. The second step builds the routing and balancing rule. The third step asks, through the selective-underfitting lens, whether the regions the model fits tightly around clean training images shrink under each loss.

Baselines

The naive reference point is a model trained with ordinary squared error on the corrupted data, using the shifted schedule of simple diffusion. This shows how much corruption leaks into generated samples when nothing is done. A model trained the same way on the uncorrupted data gives the clean ceiling. For scale, simple diffusion reports class-conditional ImageNet FID of 2.77 at 256×256 and 3.54 at 512×512. Our compute budget will be smaller, so we will reproduce a matched-budget clean ceiling rather than compare directly to those numbers. We found no reported result on Lacuna for any diffusion model trained with a robust loss on corrupted images, so the robust baselines must be run by us.

The strongest existing competitor is uniform robustness. Here the same bounded transform is applied to every band and noise level on the shifted schedule, with $\tau$ tuned by grid search. This is the direct transfer of Generalized Cross Entropy and Ghosh et al. to diffusion. A Huber-loss variant is included because it is what a practitioner would try first.

The second competitor is sample selection. Following the truncated loss idea of Generalized Cross Entropy, any sample whose total error exceeds a running quantile is dropped, with the threshold tuned.

The third competitor is a hand-tuned schedule. It uses our routing mask but fixed per-band weights found by grid search instead of the online balancer.

Two ablations isolate the key ingredients. The first is routing without balancing, which tests whether the balancer is needed at all. The second is balancing applied to uniform robustness, which tests whether routing matters or whether gradient balancing alone rescues a uniformly robust loss.

Every method gets the same data, corruption seed, architecture, number of steps, and a fixed tuning budget of the same number of short pilot runs at 128 pixels. These are then transferred unchanged to 256 and 512 pixels. The transfer is deliberate: Lucic et al. showed that settings tuned at one scale often fail at another, and transfer is the regime we care about.

Beating uniform robustness means better clean-reference FID and lower corruption leakage at 512 pixels, with the margin growing from 128 to 512. A win at 128 pixels alone would not support the claim.

Why not just…?

Why not just filter the corrupted images out first? A detector trained on known artifact types removes what it was trained to see and nothing else, and its threshold must be re-tuned for each corruption mix. Truncated sample selection is the in-training version of this idea. In classification it helped: Generalized Cross Entropy reports 54.04% versus 37.41% for cross entropy on CIFAR-100 with 60% noise. In diffusion, however, a sample's total error is dominated by high-noise steps, where clean and corrupted images look alike, so whole-sample truncation should mostly drop hard clean images. The sample-selection baseline tests this directly. We will also report its performance on a held-out corruption type absent from tuning. If filtering matches our method there, filtering is the better recommendation and we will say so.

Why not just use a robust loss everywhere and rely on the shifted noise schedule to fix global structure? This is the real threat. The schedule shift was designed for exactly the structural underfitting we worry about, and it may be enough. Our mechanism predicts the opposite, because the shift moves training toward the noise levels where a bounded loss saturates on clean images too. The uniform-robustness baseline is always run on the shifted schedule. The decisive evidence is whether the gap between uniform and routed robustness widens with resolution. If the gap stays flat or vanishes at 512 pixels, the routing claim is wrong.

Why not just hand-tune per-band weights instead of balancing online? This may well suffice, since bands are few. The hand-tuned schedule baseline exists to find out. The online balancer earns its place only if the hand-tuned weights from 128 pixels fail to transfer to 512, the brittleness Active-Passive Loss and Lucic et al. report. If they do transfer, we will drop the balancer and keep only the routing rule, which is the cheaper result.

Proposed Experiments

The main testbed is class-conditional ImageNet at 128, 256 and 512 pixels, following simple diffusion. FFHQ, used by ResDiff, serves as a second domain where facial structure makes layout failures easy to see. We inject corruption into 20% and 40% of training images, using JPEG blocking, semi-transparent watermarks, overlaid text and pasted foreign patches. One additional type, color-banding, is held out from all tuning.

We report three metrics. The first is FID against a clean reference set. The second is corruption leakage: the fraction of generated samples flagged by a separate classifier trained to detect each artifact type. The third is held-out clean denoising error per band and noise level, which directly measures where underfitting occurs.

The first decisive check is the diagnostic. Under uniform robustness, held-out clean error should rise most in coarse bands at high noise, and this rise should grow from 128 to 512 pixels. If underfitting is instead spread evenly, the premise fails.

The second decisive check is the scaling of the advantage. Routed robustness should match uniform robustness on leakage while recovering most of the clean-ceiling FID, with the FID margin increasing with resolution.

The third check concerns the balancer. When balancing is added to uniform robustness, leakage should rise back toward the squared-error level. That would show the balancer undoes robustness when applied inside the bounded bands, and that routing, not balancing, carries the effect.

The last check is the held-out corruption type. This is where filtering should fall short and a loss-based method should not, if the argument is right.