An adversarial example crafted against one image classifier often fools a different classifier that the attacker has never seen. An adversarial example here means an image changed by a tiny, nearly invisible amount so that the model gets it wrong. Separately, neural networks trained by gradient descent are known to latch onto the easiest predictive cue in the data and ignore harder ones. That cue might be a digit's color, a background, or a texture. This tendency is called simplicity bias. This note proposes that the two facts are one fact. Attacks transfer between models largely because independently trained models pick the same easy features, and an attack that breaks those features breaks every model that relies on them. If this is right, transfer becomes something we can predict and control. Two models should share attacks in proportion to how much their feature choices overlap. And an attacker who wants to reach targets that chose different features should build surrogates that deliberately learn the features ordinary training starves.
The practical stakes are concrete. The hardest open cases in transfer attacks are perturbations crafted on convolutional networks that fail on Vision Transformers (ViTs), and attacks that must force one specific wrong label rather than any wrong label. Both look like feature-selection mismatches: the target simply did not pick the shortcut that the surrogate's attack destroys. Testing this gives a mechanistic account of transfer. It also yields a cheap surrogate-side method that can be compared directly against current ensemble and model-augmentation attacks.
Core Claim
We claim that the rate at which a perturbation transfers from a surrogate model to a target model is predicted by the overlap between the features the two models rely on. Those reliance profiles are shaped by simplicity bias, and this overlap predicts transfer beyond what existing gradient-alignment measures and architecture similarity already predict. A corollary is a method we call a starvation-complement surrogate ensemble. It pairs an ordinarily trained surrogate with surrogates trained by known simplicity-bias mitigations, so that the attack is spread across both the easy feature and the starved alternatives. Because of that spread, it should transfer better to targets of unknown feature preference than an equally sized ensemble that differs only in random seeds or architectures.
What Prior Work Establishes and Leaves Open
The motivating evidence comes from three papers that touch the link without closing it. The Pitfalls of Simplicity Bias in Neural Networks shows that networks become fully invariant to complex predictive features. It also shows that, on its synthetic Linear-plus-Multi-Slab data (LMS-5), "99.6% of the adversarial 'budget' was spent attacking the linear coordinate" by a universal perturbation, meaning one perturbation applied to every input. However, it studies only white-box attacks on a single model, where the attacker has full access to that model. It never varies a surrogate's simplicity bias to see whether transfer changes. Perturbation Towards Easy Samples Improves Targeted Adversarial Transferability finds that targeted attacks transfer best when pushed toward low-loss "easy" samples of the target class, where different architectures agree most. These are plausibly exactly the samples simplicity bias fits first, but the paper never examines the surrogate's own feature simplicity. NatADiff shows that adversarial images that manipulate contextual shortcuts, such as snow for snowplows, transfer from a ResNet-50 to a ViT far better than pixel noise does. It treats shortcuts as a property of the dataset and needs about 103 seconds per image of diffusion sampling.
Two lines of work constrain the design. Adversarial Vulnerability from Interference Between Features in Superposition traces transfer to models that pack more features than they have dimensions, so features overlap and interfere, and models trained on shared data correlations end up with parallel interference patterns. It also reports that weight decay worsens vulnerability. That finding warns us that a simplicity-bias mitigation which is itself a regularizer could increase shared superposition rather than reduce shared shortcuts, so we must measure both. Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability ties transfer to how closely the two models' input gradients match. Its analysis holds in a small-perturbation regime and does not ask why gradients match. Gradient matching is therefore the explanation our feature-overlap account must beat, not merely restate.
The method draws on the simplicity-bias mitigation literature. Spectral Decoupling, from Gradient Starvation, adds a penalty on logit size so that an easy feature cannot drive the loss to zero and silence the gradients of other features. Midpoint Mixup, from Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup, mixes pairs of training images at exactly equal weight and provably keeps lagging features growing. D-BAT, from Agree to Disagree, trains models that agree on training labels but disagree on unlabeled out-of-distribution data. D-BAT evaluated "transferability" only as out-of-distribution generalization and never as adversarial transfer. We reuse these three methods unchanged as surrogate training recipes. Our contribution is to measure and exploit their effect on transfer, not to propose a new mitigation.
Research Program
The first stage is a measurement study on testbeds where the competing features are known by construction. These are LMS-5 and the MNIST-CIFAR concatenation from the Pitfalls paper, Colored MNIST, and the multi-view CIFAR-10 and CIFAR-100 variants from the Midpoint Mixup paper. On each testbed we train a population of models that vary in architecture, seed, and training recipe. The recipes are ordinary training (empirical risk minimization), Spectral Decoupling at several penalty strengths, Midpoint Mixup, uniform Mixup, D-BAT members, early-stopped checkpoints, and several weight-decay levels.
For each model we compute a feature-reliance profile. We use the Pitfalls paper's randomized-accuracy measure: shuffle one feature block across examples and record how much accuracy falls. For every ordered pair of models we then measure three things. The first is the transfer success rate of standard momentum-iterative attacks. The second is the gradient-matching quantities from Ding et al. The third is the share of perturbation energy that lands on each known feature block.
The central analysis asks whether reliance overlap explains transfer after controlling for gradient alignment and architecture. We fit transfer rate against these predictors and report the partial correlation for reliance overlap. A simple way to write the target quantity is
$$T(S \to T) \approx g\big(\langle r_S, r_T \rangle\big),$$
where $T(S \to T)$ is the fraction of surrogate-crafted examples that fool the target, $r_S$ and $r_T$ are the surrogate's and target's feature-reliance vectors, and $g$ is a monotone function we estimate rather than assume.
The second stage moves to ImageNet. There, features are not labeled, so we test predictions rather than measure reliance directly. Following ImageNet training scale is costly, so surrogates are ResNet-50 and ResNet-18 models fine-tuned from standard checkpoints under each mitigation recipe. Targets are the convolutional and ViT victims used in the transfer benchmark of Devling into Adversarial Transferability on Image Classification, evaluated on its ImageNet-compatible 1000-image set.
The starvation-complement ensemble attacks the averaged logits of one ordinarily trained surrogate plus Spectral Decoupling, Midpoint Mixup, and D-BAT surrogates, using an ordinary momentum attack. This keeps any gain attributable to the surrogates rather than to the optimizer. D-BAT needs unlabeled data from a different distribution. We use strongly augmented held-out images and acknowledge that the D-BAT paper itself flags this as fragile: if that pool shares the training shortcuts, the diversity it produces can collapse. We also test the reverse prediction. Targets fine-tuned with Spectral Decoupling should receive fewer transferred attacks from ordinary surrogates than ordinarily trained targets of the same architecture do, at matched clean accuracy.
Baselines
The naive reference point is a single ordinarily trained surrogate attacked with the momentum iterative method. For ResNet-18 to ViT, the hypothesis-space augmentation paper (Boosting Adversarial Transferability through Augmentation in Hypothesis Space) reports this at an attack success rate of "15-20%". Any mitigation-trained single surrogate must clear this to show that feature diversity matters at all.
The strongest existing methods differ by setting. For untargeted cross-architecture transfer, it is OPS from the same paper, which samples compositions of operators to explore nearby surrogate functions and reports ResNet-18 to ViT success "exceeding 75%". The benchmark review reports the Distribution Relevant Attack at "86.4%" average untargeted success and the More Bayesian Attack ensemble at "74.1%". For targeted transfer, the strongest reported result is ESMA at "51.76%" average targeted success from a ResNet-50 surrogate and "21.25%" against ViT. That is the bar for any targeted variant. We run each through the benchmark's shared protocol, with the same 1000 images, the same perturbation budget and iteration count, and the same victim list. Hyperparameters come from the original papers, and the same small grid search is applied to our recipes' penalty strengths.
The decisive fairness control is a seed-and-architecture ensemble of the same size as the starvation-complement ensemble, with every member ordinarily trained. It isolates feature diversity from mere ensembling. The Pitfalls paper's finding that ordinary ensembles do not remove simplicity bias predicts this control will transfer worse to ViTs. The second ablation swaps Midpoint Mixup for uniform Mixup, which separates the provable feature-equalizing effect from generic augmentation. This matters because the two give nearly identical multi-view CIFAR-100 error ("~53.6%" versus "~53.4%"). Beating the field means exceeding the size-matched ensemble at equal compute and matching or exceeding OPS on CNN-to-ViT transfer. Even if it falls short of OPS, the ensemble can still win as a cheap add-on, provided combining it with OPS beats OPS alone.
Why not just…?
Why not just use an early-stopped surrogate? Tsoy et al. ( Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data) show that simplicity bias sharpens as training approaches zero loss. A mid-training checkpoint may therefore carry less shortcut reliance at no cost. This is a real threat, so early-stopped checkpoints are full members of our population and a single-surrogate baseline. If they match the mitigation recipes on both reliance profiles and transfer, the honest conclusion is that "less converged" is the operative variable, and we would report it that way.
Why not just ensemble more architectures or sample more surrogate functions, as MBA and OPS do? Those methods add variety in parameters or operators, but nothing forces the members to use different features. The Pitfalls paper found that ensembles inherit the same simplicity bias. The size-matched ordinary ensemble and the OPS combination experiment settle whether feature-targeted diversity adds anything beyond them.
Why not just explain transfer by gradient alignment? Gradient matching already predicts transfer in Ding et al.'s analysis, and our overlap measure might simply be a proxy for it. The partial-correlation analysis on the controlled testbeds addresses this directly. If reliance overlap carries no predictive power once gradient alignment is included, the mechanistic claim fails, even if the ensemble method still helps.
Decisive Checks
Three outcomes would falsify the proposal. First, on the controlled testbeds, reliance overlap fails to predict transfer beyond gradient alignment. Second, perturbations from Spectral Decoupling and Midpoint Mixup surrogates do not shift energy away from the simple feature block, measured the same way as the LMS-5 "99.6%" figure. Third, on ImageNet, the starvation-complement ensemble does no better than the size-matched ordinary ensemble against ViT targets. A further check guards against the regularization paradox. If Spectral Decoupling surrogates transfer better but also show more superposition interference by the measures in Stevinson et al., the gain may come from shared interference rather than diversified features, and we would attribute it accordingly.
References
- The Pitfalls of Simplicity Bias in Neural Networks
- Perturbation Towards Easy Samples Improves Targeted Adversarial Transferability
- NatADiff: Adversarial Boundary Guidance for Natural Adversarial Diffusion
- Adversarial Vulnerability from Interference Between Features in Superposition
- Uncovering the Connections Between Adversarial Transferability and Knowledge Transferability
- Gradient Starvation: A Learning Proclivity in Neural Networks
- Provably Learning Diverse Features in Multi-View Data with Midpoint Mixup
- Agree to Disagree: Diversity through Disagreement for Better Transferability
- Devling into Adversarial Transferability on Image Classification: Review, Benchmark, and Evaluation
- Boosting Adversarial Transferability through Augmentation in Hypothesis Space
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data
- Simplicity Bias of Two-Layer Networks beyond Linearly Separable Data