Adversarial training makes an image classifier robust by training it on worst-case perturbed inputs. Finding each perturbation takes several extra forward and backward passes through the network, so robust training costs many times more than ordinary training. This note proposes moving most of that search to low resolution. If one network uses the same weights and is trained to be correct on a downsampled copy of each image as well as on the full-size image, then an attack found on a 4× smaller copy costs about 1/16 of a full-resolution step. Upsampled, that attack should already damage the full-resolution model. A few cheap coarse steps plus one full-resolution step could give a multi-step adversary at roughly the price of a single-step one.
The idea joins two lines of work that have not met. The first is cross-scale weight sharing: one operator is applied at every level of an image pyramid, as in scale-recurrent deblurring networks and scale-invariant convolution layers. The second is the effort to cut adversarial training cost, through single-step attacks, gradient recycling, and first-layer decoupling. The central claim is that scale-shared weights make adversarial search transferable across resolutions. A model that has never been asked to be consistent across scales should not get this benefit.
Core Claim
Consider a single network $f_\theta$ (weights $\theta$) that is trained so it classifies correctly when run on any level of a pyramid $x_s = D_s(x)$. Here $D_s$ downsamples by factor $s \in \{1/4, 1/2, 1\}$, and $U_s$ upsamples back. For such a network, the perturbation is built additively across scales:
$$\delta = \Pi_{\epsilon}\Big(\sum_{s} U_s(\delta_s)\Big),$$
where $\delta_s$ is found by gradient ascent on $f_\theta(x_s + \text{current perturbation at scale } s)$ and $\Pi_\epsilon$ projects onto the $\ell_\infty$ ball of radius $\epsilon$. A convolutional pass at scale $s$ costs about $s^2$ of a full pass, because convolutional FLOPs scale with image area. Taking $n_s$ steps at scale $s$ therefore costs, in full-pass units,
$$C \approx \sum_s n_s s^2 + 1,$$
where the final $+1$ is the weight update. With four steps at 1/4, two at 1/2 and one at full resolution, $C \approx 2.75$. That compares with about 2 for single-step training (Fast is better than free) and about 11 for ten-step projected gradient descent (PGD). The claim is that this seven-step pyramid adversary is nearly as strong as full-resolution PGD-7 for a scale-consistent model and much weaker for a single-resolution-trained one.
We add one further ingredient borrowed from Adversarial training for free!. That paper reuses the weight gradient from each attack step to update the model, replaying the same minibatch $m$ times. It reports that larger $m$ hurts clean accuracy because the model sees the same image repeatedly. Our coarse attack passes already produce weight gradients, on inputs that differ in resolution. Accumulating those gradients is a form of replay with built-in scale diversity. We test whether this recovers clean accuracy relative to fixed-resolution replay.
Relation to Prior Work
Three papers supply the method's pieces. Scale-Recurrent Network for Deep Image Deblurring runs one encoder-decoder with tied weights from coarse to fine. It reached 30.26 dB PSNR on GOPRO with under a third of the parameters of earlier multi-scale CNNs. Locally Scale-Invariant Convolutional Neural Networks applies tied filters across an input pyramid and pools over scales, reaching 3.13% error on MNIST-scale versus 3.48% for a standard ConvNet. FILM shares weights across pyramid levels so that small-motion training data serves large motions too. Together they show that one operator can be made valid across scales and that this sharing regularizes. None of them uses the coarse levels as a cheap place to compute gradients for an adversary. We take their weight-sharing designs and make that use central.
Three papers motivate and bound the efficiency question. Fast is better than free shows that single-step attacks with random initialization can train robust models quickly, but this approach can catastrophically overfit, meaning robustness to multi-step attacks suddenly collapses toward zero. Our method keeps a genuinely multi-step adversary, which is the usual remedy for that collapse, without paying full price for it. Adversarial training for free! recycles gradients at a fixed resolution; we recycle them across resolutions. You Only Propagate Once (YOPO) freezes the gradient at the first layer's output and iterates the attack only through that layer. Its savings are capped by how much compute the first layer represents. Our savings instead come from shrinking the whole network pass, so they do not depend on the first layer being cheap.
The closest work is Pyramid Adversarial Training Improves ViT Performance (PyramidAT). It perturbs images at pixel, 16×16-patch and 32×32-patch granularity and reports better clean and out-of-distribution accuracy for ViTs. PyramidAT computes every scale component with full-resolution passes, and it names computational cost as its main limitation. Its model does not share weights across a resolution pyramid; the pyramid lives only in the attack. We differ on both counts. Our coarse components come from coarse passes of the model itself, and the model is trained to be consistent across scales precisely so that this works. PyramidAT also constrains our design. It found that its coarse components are low-frequency shape edits, while $\ell_\infty$ robustness also needs high-frequency perturbations. This is why we always keep at least one full-resolution step rather than attacking only at coarse scale.
Research Program
We will build the method in two forms. The simple form is a standard ResNet trained on all pyramid levels, with a loss term at each scale. It is the same network run at different input sizes, made consistent by training. The structured form is a tied-branch, scale-pooled network in the style of the scale-invariant ConvNet, with a separate classification head per scale, so a coarse branch can be attacked alone.
On each minibatch, the training loop runs coarse attack steps, then medium steps, then one fine step. It optionally accumulates weight gradients from the coarse passes, then updates the weights on the full-resolution adversarial image. We will measure FLOPs and wall-clock time separately, because small images use GPUs less efficiently and FLOP savings may not translate fully into time savings.
The main diagnostic is a coarse-to-fine transfer ratio. It is the increase in full-resolution loss caused by the upsampled coarse perturbation, divided by the increase caused by full-resolution PGD with the same number of steps. We will track this ratio throughout training for scale-consistent and single-resolution models. The mechanism predicts that the ratio is high for the first and low for the second.
Baselines
The naive reference point is standard non-robust training. It fixes the cost floor and shows that robustness is near zero without adversarial training. The ceiling reference is full-resolution PGD-10 adversarial training on the same architecture, which shows how much strength the cheap adversary gives up.
The strongest competitor on cost is single-step training with random initialization from Fast is better than free. That paper reports 45% robust accuracy against PGD on CIFAR-10 in 6 minutes and 43% on ImageNet in 12 hours. We will also run its Lipschitz-regularized, weight-averaged variant (FGSM-LAW) from EasyRobust, since it was designed to resist catastrophic overfitting.
We will also compare against two gradient-reuse methods, rerun on our code and hardware. Free AT (Adversarial training for free!) reports about 46% against PGD-20 on CIFAR-10 and 40% against PGD on ImageNet ResNet-50 after 2 days on 4 GPUs. YOPO (You Only Propagate Once) reports 47.98% PGD-20 accuracy on a Wide ResNet-34 for CIFAR-10. These reported figures use PGD, not AutoAttack. For AutoAttack, no reported result was found on Lacuna for any of these baselines, so all methods will be re-evaluated ourselves.
Fairness means the same architecture, data, augmentation, epochs and learning-rate schedule family for every method. Each method gets the same hyperparameter search budget, and every method is plotted on a curve of robust accuracy against GPU-hours. Beating the strongest baseline means matching or exceeding single-step training's robust accuracy under AutoAttack at equal wall-clock time, and without collapse at any seed or tested $\epsilon$. Matching PGD-10 robust accuracy at under a third of its cost would also count.
Two ablations isolate the key ingredient. The first runs the identical coarse-to-fine attack schedule on a network trained only at full resolution. A fully convolutional ResNet can still be run on small inputs, so this tests whether scale-consistent training is what makes coarse attacks transfer. The second removes cross-resolution gradient recycling to measure its separate effect on clean accuracy.
Why not just…?
Why not just use single-step training with random initialization? Its cost of about two passes is lower than ours. On easy settings it may well match us, and that is a real threat. Our case rests on the failure mode its own authors document: catastrophic overfitting under unlucky step sizes and schedules. We will test larger $\epsilon$ values and several seeds, where single-step methods are known to be fragile, and report how often each method collapses. If single-step training never collapses in our runs and matches our AutoAttack accuracy, the efficiency claim fails. The project then reduces to its diagnostic finding about cross-scale transfer.
Why not just attack a downsampled image with an ordinary network? Any fully convolutional classifier accepts small inputs, so the weight-sharing story might seem unnecessary. The first ablation in the Baselines section tests exactly this. Our prediction is that a single-resolution network behaves inconsistently on small inputs. Its coarse gradients then point in directions that do not hurt the full-resolution model, which would show up as a low transfer ratio. If transfer turns out to be high anyway, the scale-consistency ingredient is unnecessary. We would report the simpler method as the contribution.
Why not just use Free AT with more replays? Free AT already gets multi-step-like strength from reused gradients. However, its own results show clean accuracy falling as the number of replays grows, and every replay is a full-cost pass. Our cross-resolution recycling replaces most full-cost replays with cheap, resolution-varied ones. A direct comparison at matched GPU-hours, with clean and robust accuracy reported together, settles which is better.
Why not just use YOPO? YOPO's speedup depends on the first layer being a small fraction of total compute. That share shrinks for wide stems and ViT patch embeddings. We will report YOPO's measured speedup on our ImageNet ResNet-50 alongside ours. Combining the two methods is a follow-up, since YOPO's first-layer trick can also be applied within coarse passes.
Proposed Experiments
We start on CIFAR-10 with a two-level pyramid (1/2 and full size), since 8×8 images are too small. This lets us iterate quickly and compare with the reported CIFAR numbers above. The main experiment is ImageNet-1K with ResNet-50 under the $\ell_\infty$ threat model of Fast is better than free, using the three-level pyramid. There, the area savings are large, and the 12-hour and 2-day reference runs give a natural time budget.
Robustness will be measured with AutoAttack through RobustBench. That paper documents fast defenses that look strong against plain PGD and then collapse under its attack suite. As a gradient-masking check, we will also run black-box Square Attack and confirm that accuracy falls as $\epsilon$ grows.
A secondary experiment uses ViT-B/16 on ImageNet. It asks whether pyramid perturbations generated at coarse resolution keep PyramidAT's out-of-distribution gains at a fraction of its cost. PyramidAT reports 81.71% clean accuracy, 41.42 mCE on ImageNet-C, 53.92% on ImageNet-R and 41.04% on ImageNet-Sketch.
Four checks decide the outcome. First, the transfer ratio must be clearly higher for scale-consistent models than for single-resolution ones. Second, the method must sit above single-step training and Free AT on the curve of AutoAttack accuracy against wall-clock time. Third, it must show no catastrophic overfitting at the larger $\epsilon$ values across seeds. Fourth, clean accuracy with cross-resolution recycling must exceed Free AT at a matched number of replays. If the first check fails, the mechanism is wrong. If only the second fails, the low resolution of the coarse passes underuses the GPU, and the method is useful only for high-resolution inputs.
References
- Fast is better than free: Revisiting adversarial training
- Adversarial training for free!
- Scale-Recurrent Network for Deep Image Deblurring
- Locally Scale-Invariant Convolutional Neural Networks.
- FILM: Frame Interpolation for Large Motion
- You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle
- Pyramid Adversarial Training Improves ViT Performance
- RobustBench: a standardized adversarial robustness benchmark