Adversarial training makes a network robust to small, deliberately crafted input perturbations. It works by attacking the model on every batch and then training on the attacked inputs. The attack step multiplies training cost many times over. A single careful ImageNet study of this kind consumed about 20,000 GPU-hours (Revisiting Adversarial Training for ImageNet), and its biggest lever was architectural: swapping a vision transformer's patch-cutting input layer for a small convolutional one moved robust accuracy by several points. Practitioners therefore face an expensive choice before training starts: which backbone family is worth adversarially training? Cheap "zero-cost proxies" exist to answer this. They score an untrained network from a few forward and backward passes. But they have only been validated within narrow, homogeneous search spaces. A separate literature on transfer learning shows that model-derived similarity scores routinely break when the comparison crosses architecture families, for example from convolutional nets to transformers.
This note proposes to test, and then repair, cheap robustness proxies across architecture families. The hypothesis is mechanistic. Proxies fail across families because part of what they measure lives in each network's internal feature space, and those spaces are not comparable between a convolutional net and a transformer. Input pixels and output predictions are shared by every architecture. A proxy built only from quantities measured in input and output space should therefore rank convolutional nets, transformers, and hybrids on a common scale. The deliverable is a measured curve of ranking accuracy against compute for every proxy type. It tells a practitioner how much compute to spend predicting adversarial-training outcomes before committing to a full run.
Core Claim
CRoZe scores an architecture at initialization by how consistently it behaves on clean versus perturbed inputs. It combines three signals: internal feature similarity, the distance between one-step parameter updates, and gradient alignment. CRoZe reports a Spearman correlation of about 0.97 with fully trained robust accuracy, but only inside the DARTS and NAS-Bench-201 cell spaces. This is the result that motivates the work. It shows robustness potential can be read off cheaply.
Transfer-learning work shows why the result may not travel to other families. Fisher-based task similarity predicts well for BERT but degrades for RoBERTa (What to Pre-Train on?). Similarity measures are tied to the exact architecture and initialization they came from (Ma et al.). Transferability predictors break between convolutional nets and vision transformers (Tan et al.). These papers constrain the design: any feature-space term is suspect across families.
The method comes from IGAM, which transfers robustness between very different networks by matching input gradients. Its argument is that pixel space is common to every architecture. We reuse that argument for measurement rather than training.
Our claim is twofold. First, CRoZe's ranking accuracy will drop sharply on a cross-family pool, and the drop will be traceable to its feature-space term. Second, a proxy restricted to input-space and output-space quantities will recover cross-family ranking comparable to a short adversarial-training "probe run," at a small fraction of that run's cost.
Research Program
The proposed proxy evaluates a network $f$ on a few minibatches. For each input $x$, take a one-step attack $\delta$ of size $\epsilon$. Compute three quantities, all living in pixel or prediction space:
- the cosine alignment between the input gradients at $x$ and at $x+\delta$;
- a local-linearity gap, defined as the extra loss a two-step attack finds beyond a one-step attack;
- the divergence between the predicted class distributions at $x$ and $x+\delta$.
The score is
$$S(f) = \mathbb{E}_x\big[\cos(\nabla_x \mathcal{L}(x), \nabla_x \mathcal{L}(x+\delta))\big] - \lambda_1\,\mathbb{E}_x\big[\mathcal{L}_{2\text{-step}} - \mathcal{L}_{1\text{-step}}\big] - \lambda_2\,\mathbb{E}_x\big[\mathrm{KL}(p(x)\,\|\,p(x+\delta))\big],$$
where $\mathcal{L}$ is cross-entropy, $p$ is the softmax output, and $\lambda_1, \lambda_2$ are weights. The weights are fit on one architecture pool and frozen before evaluation on held-out pools.
The local-linearity term is motivated by the finding in Why Does Little Robustness Help? that smoothness and gradient similarity govern how perturbations behave across architectures. That paper studies attacks; we use the same quantities as a defensive diagnostic.
The proxy is computed in two regimes:
- At random initialization, matching CRoZe's setting.
- On a clean-pretrained checkpoint. Modern adversarial training usually starts from pretrained weights, and the ImageNet recipe in Revisiting Adversarial Training depends on such starting points.
The core analysis decomposes CRoZe into its feature, parameter, and gradient terms. It measures how much each term contributes to within-family versus cross-family ranking. This turns the claim that "proxies break across families" into a specific diagnosis of which term breaks.
Ground Truth and Benchmarks
Ranking studies need many fully trained robust models under one fixed recipe. RobustBench models were trained with heterogeneous recipes, so they cannot serve as clean labels for architecture effects. We therefore build the ground truth on CIFAR-10. We train about 30 architectures under a single adversarial-training recipe. The pool spans four groups:
- ResNets and Wide-ResNets;
- small ConvNeXts;
- small ViTs in patch-stem and convolutional-stem variants;
- a sample of DARTS cells.
Robust accuracy is evaluated with AutoAttack through the RobustBench tooling. That benchmark documents robustness overestimates exceeding 10% under weaker attacks, so naive PGD numbers are unacceptable as labels.
Two further pools test generalization:
- NAS-Bench-201 serves as a within-family control that should reproduce CRoZe's strong correlation.
- ImageNet provides a small out-of-sample check. Here we rank only the ConvNeXt and ViT, patch-stem and convolutional-stem configurations studied in Revisiting Adversarial Training, using their trained checkpoints and reported AutoAttack results.
We also test transfer of the ranking to threat models the proxy never saw, namely $\ell_2$ and $\ell_1$ attacks. That paper found convolutional-stem ViTs generalize better to unseen threat models, which gives a qualitative target the proxy should recover.
Metrics are Spearman and Kendall rank correlation against AutoAttack robust accuracy. We also report a top-3 regret borrowed from What to Pre-Train on?: the robust-accuracy gap between the best of the proxy's top three picks and the true best architecture. All of these are plotted against the proxy's GPU-hours.
Baselines
The naive reference point is to rank architectures by clean accuracy or parameter count. This is what practitioners do by default. Any proxy that fails to beat it on top-3 regret is not worth computing.
The strongest existing zero-cost method is CRoZe. It is run with the authors' three-term definition, the same minibatches, the same perturbation size, and the same number of forward and backward passes as our proxy. The comparison therefore isolates what is measured rather than how much compute is spent. The reported reference is Spearman ρ ≈ 0.97 with fully trained models on the DARTS space (CRoZe). We expect to reproduce this within families. No reported result was found on Lacuna for CRoZe on cross-family pools of ConvNeXts and ViTs. Beating CRoZe means a statistically higher cross-family Spearman correlation, by bootstrap over architectures, without losing within-family correlation.
The strongest competitor overall is a short adversarial-training probe. We train each architecture for a small budget using ATTA-1. ATTA-1 is the cheapest established method: it recycles one-step perturbations across epochs and reports a 14.1× speedup over PGD-40 on CIFAR-10 (ATTA). We then rank architectures by their resulting robust accuracy. We sweep the probe budget from roughly 0.1% to 5% of a full run, so the comparison is a curve rather than a single point. No reported result was found on Lacuna for the ranking accuracy of such truncated runs.
Our proxy "beats" the probe if, at equal GPU-hours, it has higher rank correlation. It also beats the probe if it matches the probe's correlation at one-tenth or less of the probe's compute.
The key ablation is a set of mixed proxies, each run under the same budget:
- CRoZe with its feature term deleted;
- our proxy with CRoZe's feature term added back;
- each of our three terms alone.
If the feature term is what breaks cross-family ranking, deleting it should help CRoZe and adding it should hurt ours.
Why not just…?
Why not just run a short adversarial-training probe on each candidate? This is a real threat, and it may win. Early training curves contain information no initialization-time score can. The budget sweep is designed to settle the question honestly. If the probe at a matched tiny budget ranks better than any zero-cost score, the paper's recommendation becomes "spend X% of a run on ATTA-style probes." The diagnostic of which proxy terms fail across families still stands. The proposal only claims an advantage at budgets where the probe has not yet separated architectures.
Why not just use CRoZe as is? Its 0.97 correlation was measured only in cell-based convolutional spaces. Three transfer-learning studies show model-internal similarity measures lose their meaning across families (What to Pre-Train on?, Ma et al., Tan et al.). The term-by-term ablation tests this directly. If CRoZe holds up across families, that is itself a useful negative result for our hypothesis, and we report it.
Why not just pick whatever architecture tops RobustBench, or follow the convolutional-stem prescription? Leaderboard entries mix architecture effects with recipe, data, and compute differences, so a leaderboard rank does not predict what a new backbone will gain under one's own recipe. Where to start? found the analogous lesson in NLP: off-the-shelf checkpoints mostly underperformed base models, and rankings changed across target architectures. The ImageNet check asks whether the proxy recovers the convolutional-stem advantage without being told about it. If it does, the proxy encodes more than a single rule of thumb.
Why not just rank by clean accuracy? Clean and robust accuracy are known to trade off. In the ImageNet study, the stem change shifted robustness substantially, which a clean-accuracy ranking need not reflect. The naive baseline is included precisely so this objection gets a number rather than an argument.
Decisive Checks
Three outcomes settle the proposal. First, on NAS-Bench-201, CRoZe and our proxy should both show high correlation; failure here means our replication is broken. Second, on the cross-family CIFAR-10 pool, CRoZe's correlation should drop, and removing its feature term should recover most of the drop. Third, on the ImageNet check, our proxy's top-3 regret should beat clean-accuracy ranking, and on the CIFAR-10 pool its correlation should be at least comparable to ATTA-1 probes of equal cost.
If the second outcome fails, the input-space hypothesis is wrong. If the third fails, short training probes are the answer. In either case the measured compute–accuracy curve still tells practitioners how to spend a 20,000 GPU-hour budget more wisely.
References
- Revisiting Adversarial Training for ImageNet: Architectures, Training and Generalization across Threat Models
- Generalizable Lightweight Proxy for Robust NAS against Diverse Perturbations
- What to Pre-Train on? Efficient Intermediate Task Selection
- GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks
- Finding the Most Transferable Tasks for Brain Image Segmentation
- What It Thinks Is Important Is Important: Robustness Transfers Through Input Gradients
- Why Does Little Robustness Help? Understanding Adversarial Transferability From Surrogate Training
- RobustBench: a standardized adversarial robustness benchmark
- Efficient Adversarial Training With Transferable Adversarial Examples
- Where to start? Analyzing the potential value of intermediate models