Transferable Attacks from the Signal Part of Surrogate Weights

An adversarial example crafted on a model you own (the "surrogate") often fools a different model you cannot see (the "target"). The best-known way to make these attacks transfer well is to give the surrogate a little robustness, either through light adversarial training or through second-order smoothness regularizers. Both options add substantial training cost. We propose a cheaper route with a sharper explanation behind it. A trained weight matrix can be split into a low-rank "signal" update that captures features shared across models, and a residual "noise" update that memorizes quirks of this particular run: its random initialization and its specific training samples. Surrogate gradients that pass through the noise part are private to that model and should not line up with any target's gradients. If so, deleting the noise part of a normally trained surrogate after training, with one singular-value decomposition per layer, should recover much of the transfer benefit of robust surrogates at close to zero extra cost.

In two sentences: the part of a surrogate's input gradient that fails to transfer comes from the weight components that memorize noise, and those components can be located from the singular-value spectrum of each layer's learned update. Removing them gives a surrogate that is smooth where it counts while staying faithful to the data, and it needs no adversarial training at all.

Core Claim

Write each layer's final weights as

$$W_T = W_0 + S + N,$$

where $W_0$ is the random initialization, $S$ is the part of the learned update $\Delta W = W_T - W_0$ carried by singular directions whose singular values sit above the edge of the random-matrix bulk, and $N$ is the remaining bulk of $\Delta W$. This is the "spiked random matrix" picture from Cui et al. and Yoo et al.. Those papers model trained weights as a random baseline plus an emerging low-rank spike aligned with the target features, and we reuse their decomposition as the method. Our "denoised surrogate" replaces $W_T$ with $W_0 + S$ in every layer, possibly with a lightly shrunk $W_0$, and then recalibrates batch-norm statistics on clean data.

The claim has three testable parts. First, input gradients of two independently trained models are aligned mainly through their $S$ parts, and the contribution flowing through $N$ is nearly orthogonal across runs. Second, removing $N$ lowers the surrogate's input-gradient roughness without moving it off the data distribution, so it raises model smoothness and gradient similarity together. Third, part of the falling side of the bell-shaped "little robustness" curve comes from adversarial training at larger budgets inflating off-manifold weight components. If so, denoising large-budget adversarially trained surrogates should partially flatten that falling side.

Why This Is Different From What Exists

Four lines of work motivate the claim. Why Does Little Robustness Help? shows that transferability trades model smoothness against gradient similarity, and that small-budget adversarial training or Sharpness-Aware Minimization (SAM) with input-gradient regularization yields strong surrogates. It never asks which weight components cause the loss of similarity, and its best recipes need second-order gradients. Effect of Ambient-Intrinsic Dimension Gap on Adversarial Vulnerability proves, for two-layer ReLU networks, that off-manifold "volatile" weight components drive vulnerability. It studies white-box attacks on a single model and says nothing about transfer. Han et al. and Tang et al. show that training first learns shared signal and later memorizes sample-specific noise, and that optimizers such as Adam speed up the noise growth. That is the mechanism behind our prediction that noise components are idiosyncratic.

Two further works supply related techniques that we deliberately do not copy. Why Does Pruning during Training Work? shows that magnitude pruning during training filters out noise-memorizing coordinates. Its target is clean generalization, and it prunes coordinates rather than spectral directions. Subspace Adversarial Training confines fast adversarial training to a low-dimensional subspace of the weight trajectory. That subspace is found empirically from early checkpoints, and the method targets white-box defense, not surrogate quality.

The closest competitor at attack time is Adversarial Weight Tuning. It perturbs surrogate weights during the attack to find flat maxima, which roughly doubles backward passes per attack, and it treats the weight change as a flatness objective with no signal/noise structure. We differ from all of these in three ways. We act on the surrogate once, after ordinary training. We choose what to delete using a random-matrix criterion applied to the learned update rather than to the raw weights. We evaluate on black-box transfer.

Research Program

The first stage is diagnostic. On CIFAR-10, CIFAR-100 and ImageNette, we train families of surrogates (ResNet-18, VGG-16, DenseNet, and a small ViT) from saved initializations with five seeds each. For every layer we compute the spectrum of $\Delta W$ and check whether a clean spike and bulk separation exists. The bulk edge is estimated by fitting a Marchenko–Pastur law to the lower singular values. The separation is not guaranteed in deep networks, and if it fails the whole approach reduces to picking a rank by hand, which is itself a finding.

We then split each model's input gradient into the part flowing through $S$ and the part flowing through $N$, using a first-order expansion around $W_0 + S$. For each part we measure cosine similarity with the gradients of other seeds and other architectures. The prediction is specific: cross-seed similarity of the $N$-path gradient should be near zero, and should stay near zero longer into training than similarity of the $S$-path gradient, given that noise memorization comes late (Han et al.).

The second stage is the intervention. We build denoised surrogates and attack a pool of independently trained targets with standard iterative transfer attacks. The ImageNet-scale test uses one ResNet-50 we train ourselves with a saved initialization, attacking public ImageNet models on the ImageNet-compatible 1,000-image set used by AWT. For publicly released models whose initialization is unknown, we test a fallback that uses the spike/bulk split of $W_T$ itself. We also test a fine-tuning variant in which $W_0$ is the pretrained backbone. That variant connects to Two Sides of the Same Coin, which shows fine-tuned models inherit their backbone's geometry. Compute cost is logged as wall-clock time relative to one standard training run.

The third stage tests the bell curve. We train small-budget adversarially trained surrogates across a sweep of perturbation budgets, using single-step training at small budgets to keep cost down. We then denoise each one and ask whether the high-budget drop in transfer shrinks. This addresses the open tension between the two lines of work: whether the similarity loss under heavier adversarial training is carried by weight components that denoising can remove.

Baselines

The naive reference point is a standard, cleanly trained surrogate of the same architecture, attacked with the same iterative method, step count and perturbation budget. Any gain we report is measured against it first.

The strongest existing surrogate-side competitor is the SAM plus input-gradient-regularization or Jacobian-regularization surrogate from Why Does Little Robustness Help?, along with its small-budget adversarially trained surrogates. That paper reports near 100% untargeted attack success on CIFAR-10 against ResNets, VGG and DenseNets. This ceiling means untargeted CIFAR-10 cannot discriminate between methods. Our primary metrics are therefore targeted success, CNN-to-ViT and ViT-to-CNN transfer, and CIFAR-100 and ImageNette, where no reported number was found on Lacuna for these baselines. We rerun their recipe ourselves with the same architectures, data and epochs, and we give every method an equal hyperparameter search budget.

The strongest attack-side competitor is AWT. It reports roughly a 10% average attack-success gain on Transformer targets, about 5% on CNNs, and an increase from 49.7% to 61.3% on Inception-v4 when combined with Admix, all on the ImageNet-compatible set (Adversarial Weight Tuning). We run AWT on our standard surrogate and on our denoised surrogate, which tests both substitution and complementarity.

Beating the field means one of two outcomes. The first is matching the SAM+regularization surrogates within two points of attack success at under 5% of their extra compute. The second is exceeding them when denoising is stacked on a small-budget surrogate, while also improving on AWT's reported Inception-v4 figure under the same attack.

Four ablations isolate the key ingredient, which is spectral selection applied to the learned update. The first truncates the singular values of $W_T$ instead of $\Delta W$, which amounts to ordinary low-rank compression. The second removes a random set of singular directions of $\Delta W$ with matched energy. The third applies magnitude pruning to $\Delta W$ at matched sparsity, in the spirit of Why Does Pruning during Training Work?. The fourth keeps $S$ but also removes $W_0$. The claim survives only if denoising $\Delta W$ spectrally beats all four.

Why not just…?

Why not just early-stop the surrogate before it memorizes noise? This is the most serious threat. Han et al. suggests early stopping catches the model after signal learning and before noise memorization, which is exactly the state we try to reach after the fact. The difference is that early stopping also stops signal growth in layers that are still learning, while post-hoc denoising keeps the fully trained signal. We treat early-stopped surrogates at every checkpoint as a baseline curve. If the best early-stopped checkpoint matches the denoised final model, we will report that the mechanism is right but the intervention is unnecessary.

Why not just use a slightly robust surrogate? It works, but it costs adversarial or second-order training, and at higher budgets it moves the surrogate off the data distribution. Our third stage tests whether denoising adds to small-budget training rather than competing with it. If denoising gives nothing on top of the best small-budget surrogate, the transfer claim weakens to a cost-saving claim.

Why not just compress the surrogate with truncated SVD or pruning? Compression discards the smallest directions of the whole weight matrix, including structure inherited from $W_0$. It also says nothing about which directions are idiosyncratic. The $W_T$-truncation and magnitude-pruning ablations answer this directly. There is also a caution in the literature: Publishing Efficient On-device Models shows compressed models transfer well to their own parent, which is transfer within one lineage, not across independent models. We therefore evaluate only on independently trained targets.

Why not just use AWT at attack time? AWT flattens the surrogate locally for every attack, at roughly twice the backward cost per example. Denoising is a one-time change. If AWT on a standard surrogate beats AWT on a denoised one, denoising adds nothing, and the experiment above measures exactly that.

Proposed Experiments and Decisive Checks

The first decisive check is the gradient decomposition. The mechanism is supported if the cosine similarity of $N$-path gradients across seeds is near zero while the $S$-path similarity is clearly positive, and if denoised surrogates show higher gradient similarity with targets and lower input-gradient norms than standard surrogates. If $N$-path gradients turn out to be as transferable as $S$-path gradients, the idea is falsified regardless of attack-success numbers.

The second check is the cost-matched transfer comparison against the SAM+regularization surrogates. The measures are targeted and cross-family success on CIFAR-100, ImageNette and the ImageNet-compatible set, reported alongside clean accuracy and wall-clock cost.

The third check is the budget sweep. It tells us whether volatile weight growth explains the falling side of the little-robustness curve, which neither Why Does Little Robustness Help? nor Effect of Ambient-Intrinsic Dimension Gap on Adversarial Vulnerability settles. Training runs use easyrobust for its adversarial and fast adversarial training components. Attack convergence is audited with IndicatorsOfAttackFailure, so that no transfer gain is an artifact of a weaker attack on the baseline.

A positive result would have a defensive corollary, studied only as a secondary measurement. Removing $N$ from a deployed model should not change which of its adversarial examples transfer to other models, and that prediction can be checked against the in-transferability measure used in Q-TART.