Adversarial training is the main way to make an image classifier resist adversarial examples, which are small, deliberately crafted input changes that flip its prediction. It is expensive because every training step first runs an attack to find the worst nearby input, usually with several gradient steps, and only then updates the model. Two recent scaling studies reach very pessimistic conclusions about this cost. One finds robust accuracy grows with training compute at an exponent of only about 0.01. The other extrapolates that matching human robustness on CIFAR-10 would take $10^{30}$ FLOPs. Both fit curves to total compute. Total compute, though, mixes two different purchases: compute spent learning (more parameters, more data) and compute spent attacking (more inner attack steps per example). Neither study's curve has a ceiling either, even though humans themselves top out near 90% on these perturbed images.
We propose a scaling law for adversarial training with two features. The number of inner attack steps is an explicit axis, priced in FLOPs. Robust error has an irreducible floor. We test the law the way scaling-law extrapolation should be tested: fit it on small runs, then predict held-out runs an order of magnitude larger before seeing them. The practical question is whether the pessimistic projections come from how compute was allocated and how the curves were fitted, rather than from a property of robustness. A second question follows from it: does the best number of attack steps per example change as models grow? If it does, recipes tuned on small models are systematically wrong at scale.
Core Claim
Write $N$ for model parameters, $D$ for the number of training examples seen, and $K$ for the number of inner attack steps. Training cost is then roughly $C \approx c\,N D (K+1)$, because each example costs one forward-backward pass per attack step plus one for the weight update. We hypothesize that an example attacked with $K$ steps is worth only a fraction $h(K)\le 1$ of a fully adversarial example. Robust error $E$, measured by AutoAttack, would then follow
$$E(N,D,K) = E_\infty + A N^{-a} + B\,\big(D\,h(K)\big)^{-b}, \qquad h(K) = \frac{K}{K+\kappa}.$$
Here $E_\infty$ is the irreducible error floor, $A, a, B, b$ are fitted constants, and $\kappa$ is the number of steps at which a training attack reaches half its full value. The saturating form of $h$ encodes the familiar observation that gains from extra attack steps flatten after a handful.
At fixed compute, this law makes a sharp prediction. If $h$ does not depend on $N$, the optimal $K$ is the same at every scale, because the best choice maximizes $h(K)/(K+1)$ whatever the budget. If larger models need stronger training attacks (a $\kappa$ that grows with $N$), the optimal $K$ drifts upward with scale, and total-FLOP curves fitted at a fixed $K$ will bend in ways a single power law cannot capture. Which case holds is an empirical question that no paper in our file answers, and the answer decides how compute should be split at scale.
Relation to Prior Work
Two papers motivate the proposal and supply the baselines to beat. Scaling Compute Is Not All You Need for Adversarial Robustness trains 320 WideResNets, varying the number of inner steps from 1 to 10, and fits accuracy as a power of total FLOPs with exponent about 0.01 ($R^2 = 0.76$). It treats attack steps as one more source of FLOPs and does not separate them from learning compute. Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies fits robust loss jointly in $N$, $D$ and synthetic-data quality. With that law it reaches 74% AutoAttack accuracy and projects the $10^{30}$-FLOP figure. Its human study also suggests a ceiling near 90%. We keep its synthetic-data setting but add $K$ as an axis and $E_\infty$ as a term, and we score laws by how well they predict held-out large runs.
Two papers constrain how we fit. (Mis)Fitting Scaling Laws shows that unreported choices about aggregation, outlier exclusion and fitting objective change frontier predictions substantially. We therefore pre-register one fitting protocol and report sensitivity to each choice. Broken Neural Scaling Laws offers flexible piecewise curves and serves as a competing functional form.
The backtesting method comes from Pretraining Scaling Laws for Generative Evaluations of Language Models, which checks laws fitted on tiny models against larger already-trained ones. Two further papers warn that hyperparameters tuned small do not transfer. Scaling Optimal LR Across Token Horizons finds the best learning rate falls as training lengthens. The Effect of Network Width on Stochastic Gradient Descent and Generalization finds wider networks need different gradient-noise settings. The robustness scaling studies above held such settings largely fixed. The roughly 3.6% reproduction gap reported in the 320-model study is a warning that this can distort curves.
Research Program
All core experiments use CIFAR-10 under $\ell_\infty$ perturbations of radius $8/255$, scored with AutoAttack through RobustBench. The models are WideResNets of increasing width and depth, trained with TRADES (an adversarial training loss that trades clean accuracy against robustness). Training data is CIFAR-10 plus synthetic images from a single fixed EDM diffusion generator. Holding the generator fixed keeps data quality constant, so data quality cannot be confused with the $D$ axis.
The sweep is a grid over $N$, $D$ and $K \in \{1,2,3,5,7,10\}$, spanning about three orders of magnitude of compute. At every $(N,D)$ cell we run a small learning-rate grid. We also rescale the PGD step size with $K$, so that short attacks are not handicapped by a step size tuned for long ones. Each fitted point therefore reflects tuned training, not a fixed small-model recipe.
The law is fitted only on runs in the lower two decades of compute. We then predict, before training, the robust error of runs in the top decade and the optimal $K$ at those budgets.
Proxy runs are evaluated on a fixed 1,000-image AutoAttack subset. All held-out large runs use the full test set. Single-step and low-$K$ runs are at risk of gradient masking, where a model hides its gradients and appears robust without being robust. For these runs we add diagnostics from IndicatorsOfAttackFailure. We exclude a run only when those diagnostics flag attack failure, and we report every exclusion.
A secondary arm asks whether parameter-efficient adversarial fine-tuning simply shifts the constants of the law. It uses FullLoRA-AT, which trains low-rank adapters and normalization layers on a frozen Vision Transformer. Its attacks still backpropagate through the whole backbone, so it should mainly change the per-step cost $c$, not $h(K)$. We test this with the same $K$ sweep on CIFAR-10 at two backbone sizes.
Baselines
The naive reference point is the total-FLOP power law from Scaling Compute Is Not All You Need for Adversarial Robustness, refit on our lower-decade runs. That paper reports an exponent of about 0.01 with $R^2 = 0.76$ in-sample. It is the right floor because it is the law behind the "compute is not enough" conclusion. It receives the same runs, the same fitting objective and the same held-out evaluation as every other law.
The strongest existing method is the joint $N$, $D$ law of Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies. Because we hold synthetic-data quality fixed, it reduces here to $N$ and $D$ terms at the authors' fixed $K$. It is the strongest competitor because it already produced compute-optimal allocations that beat earlier models, reaching 74% AutoAttack accuracy on CIFAR-10. We found no reported held-out extrapolation error for this law on Lacuna.
We fit this law two ways. The first uses only our runs at its native single $K$. The second fits it to all runs while ignoring $K$, which tests whether simply pooling over $K$ is enough. Our law beats it if it achieves lower held-out error in the top decade, by more than the spread across training seeds. Beyond prediction, the allocation our law recommends, once actually trained, must reach higher AutoAttack accuracy at equal FLOPs than the allocation the $N$, $D$ law recommends.
A flexible-curve baseline fits Broken Neural Scaling Laws to total FLOPs. This tests whether added curvature alone, without structure, matches our accuracy.
We also include the gradient-boosting regressor of the 320-model study, which reports $R^2 = 0.92$ when predicting AutoAttack accuracy from configuration. We expect it to interpolate well and extrapolate poorly.
Three ablations isolate the ingredients of our law. The first removes $E_\infty$. The second removes the $K$ axis by setting $h \equiv 1$ and adding $K$ only to cost. The third refits on runs trained with fixed, small-model hyperparameters instead of retuned ones.
Why not just…?
Why not just refit existing curves with a ceiling? This is the most serious threat. Adding $E_\infty$, whether fitted freely or fixed near the human error of about 10%, may by itself remove most of the extrapolation error and deflate the $10^{30}$ projection. The ablation that keeps $E_\infty$ but drops the $K$ axis tests exactly this. If it matches the full law on held-out error and on optimal-allocation accuracy, we will say so. The contribution then becomes a ceiling-aware and properly backtested refit, and the claim about attack steps is dropped.
Why not just predict from configurations with a learned regressor? The 320-model study's regressor reaches $R^2 = 0.92$, but it was evaluated inside the range of configurations it was trained on. Tree models predict a constant beyond their training range, so they cannot project to larger budgets. Our top-decade holdout measures this failure directly instead of assuming it.
Why not just use cheap single-step or parameter-efficient adversarial training and skip the analysis? Cheaper attacks are not an alternative to the law. They are the $K=1$ point on its attack axis, and the question is whether their savings are worth the lost attack strength at a given scale. Parameter-efficient methods such as FullLoRA-AT report about 75% of full fine-tuning time. That is a constant-factor saving, because the inner attack still runs through the whole backbone. Our secondary arm checks whether such methods change only the per-step cost constant or also the shape of $h(K)$.
Why not just add more synthetic data? Data enters as the $D$ axis. The 500-million-image study already shows that generator quality limits returns. We hold the generator fixed so that data quality cannot be mistaken for a $K$ effect.
Decisive Checks
The program succeeds if three things hold. First, our law, fitted on the lower two decades, predicts top-decade AutoAttack error better than both the total-FLOP law and the $N$, $D$ law. Second, the ablations show that both $E_\infty$ and the $K$ axis contribute. Third, the optimal $K$ either stays constant or drifts with $N$ in the direction the fitted $\kappa(N)$ predicts, and the predicted allocation, once trained, wins at equal FLOPs.
The program fails cleanly in two ways. If the $K$ axis adds nothing beyond the ceiling, the attack-step claim is wrong. If no law extrapolates better than the total-FLOP power law, the pessimistic exponent was right. Either way, we will report the result with the projected FLOPs to reach 90% recomputed under each fitted law, so readers can see how much of the $10^{30}$ figure survives.
References
- Scaling Compute Is Not All You Need for Adversarial Robustness
- Adversarial Robustness Limits via Scaling-Law and Human-Alignment Studies
- (Mis)Fitting Scaling Laws: A Survey of Scaling Law Fitting Techniques in Deep Learning
- Broken Neural Scaling Laws
- Pretraining Scaling Laws for Generative Evaluations of Language Models
- Scaling Optimal LR Across Token Horizons
- The Effect of Network Width on Stochastic Gradient Descent and Generalization: an Empirical Study
- RobustBench: a standardized adversarial robustness benchmark
- FullLoRA-AT: Efficiently Boosting the Robustness of Pretrained Vision Transformers