Making a vision model both small and hard to fool usually costs three expensive jobs in sequence. First you train a dense model. Then you adversarially train it, which means repeatedly attacking the model during training and learning from the attacks. Then you compress it and fine-tune again. Adversarial training is the dominant cost because every weight update needs several extra forward and backward passes to construct the attack. This proposal asks a simple question. If the deployed model will drop most of its image patches anyway, why should the attack that trains it be computed on the full, dense network? We propose to train a Vision Transformer (ViT, a model that splits an image into patches, or "tokens," and processes them with attention) to learn its input-dependent token-dropping rule and its adversarial robustness in a single training pass. The attack is generated through the same pruned computation the model will use at deployment, so each attack step costs only as much as an inference on the compressed model.
The second ingredient addresses a security hole specific to dynamic compression. When a model decides per input how many tokens to keep, an attacker can target that decision. Prior work shows that attacks inflate attention scores so the selector keeps poisoned tokens and discards clean ones. An attacker could also push the model into its most expensive execution path. We therefore charge the compute penalty on worst-case inputs rather than average ones. The inner attack maximizes both the classification loss and the compute spent, and the training objective penalizes compute measured on those attacked inputs. The result is a single recipe in which the resource budget, the compression schedule, and the cost of adversarial training are all handled by one optimization.
Core Claim
Our hypothesis has two parts. First, attacks constructed on the hard-pruned forward pass, with straight-through gradients for the token selector, are strong enough to train a robust pruned ViT. Its robust accuracy under AutoAttack should match a model trained with dense-path attacks at the same final compute. The adversarial training cost would then shrink by roughly the model's inference speedup. Second, adding a compute term to the attacker's objective stops adversarial inputs from inflating inference cost, and it reduces attention hijacking of the selector, without the clean-accuracy collapse that robustness-aware token selection has shown. Each part is falsifiable separately. The first fails if sparse-path attacks cause gradient masking, meaning attacks that look strong during training but are weak against a model whose gradients have become uninformative. The second fails if the compute term buys nothing over a hard cap on kept tokens.
What Exists and Where the Gap Is
The closest single-stage work is Adversarial Robust Model Compression Using In-Train Pruning. It learns weight and channel masks from scratch under fast single-step adversarial training with a multiply-accumulate penalty. It is limited to CNNs, evaluates against basic attacks rather than AutoAttack, and computes its attacks on the full masked network. HYDRA learns robust pruning masks by minimizing the adversarial loss. It starts from a robustly pre-trained dense checkpoint, so it keeps the multi-stage cost. These two papers motivate our single-pass framing. Neither touches tokens, and neither uses compression to cheapen the attack itself.
On the efficiency side, we reuse methods from two papers. Adaptive Sparse ViT supplies the token selector. It scores tokens from attention head outputs, learns per-layer thresholds through a straight-through estimator (STE, which applies a hard keep/drop decision in the forward pass but passes gradients as if the decision were smooth), and adds a FLOPs penalty. It considers only clean data. Differentiable Joint Pruning and Quantization supplies the template of a differentiable hardware-cost penalty inside one training pass, again only on clean data.
Two papers motivate the robustness concern. On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression shows that token compression acts as a "distortion concentrator": attacks raise attention on perturbed tokens so the selector keeps them. It also finds that robustness-aware selection trades away clean accuracy. Attacking Compressed Vision Transformers reports that more aggressive sparsification makes ViTs more vulnerable. Neither proposes a training method.
Two papers constrain the design. Adversarial Pruning: A Survey and Benchmark finds, on CNNs under AutoAttack, that how the budget is allocated across layers matters more than how individual weights are scored. It also finds that published robust accuracies were overstated by up to 6.37%. Its first finding forces us to test a fixed-ratio budget against our dynamic one. Its second, together with AutoAttack, forces adaptive evaluation. SAFE improves pruned robustness by seeking flat minima, but its sharpness-aware step doubles gradient cost. That is the opposite of our goal, so we exclude flat-minimum terms.
Research Program
We fine-tune a clean pretrained DeiT checkpoint in one pass. Learning the token thresholds and the robust weights happens together, so there is no separate robust pre-training, pruning stage, or recovery fine-tune. Let $f_{\theta,\tau}$ be the ViT with weights $\theta$ and per-layer keep thresholds $\tau$. Let $R_\tau(x)$ be the FLOPs spent on input $x$ given which tokens survive. Let $B$ be the target budget, and let $\epsilon$ be the $\ell_\infty$ perturbation radius. Training solves
$$\min_{\theta,\tau}\ \mathbb{E}_{(x,y)}\Big[\mathcal{L}\big(f_{\theta,\tau}(x+\delta^\star),y\big)+\lambda\,\big(R_\tau(x+\delta^\star)-B\big)_+\Big],\qquad \delta^\star\approx\arg\max_{\|\delta\|_\infty\le\epsilon}\ \mathcal{L}\big(f_{\theta,\tau}(x+\delta),y\big)+\gamma\,R_\tau(x+\delta),$$
where $\mathcal{L}$ is cross-entropy, $\lambda$ weights budget violation on attacked inputs, and $\gamma$ is the attacker's incentive to inflate compute. Setting $\gamma=0$ and evaluating the budget term on clean inputs recovers "adversarial training of Adaptive Sparse ViT."
The inner maximization uses $K$ steps of projected gradient ascent (PGD). Each step runs the hard-pruned forward and backward pass, so dropped tokens are never computed past the layer where they are dropped. A dropped patch's pixels can affect the output only through the selector's scores. Gradients reach those pixels through the selector's STE, which preserves the attacker's ability to hijack token ranking. A per-layer hard cap on the number of kept tokens bounds worst-case compute regardless of $\gamma$. Inside that cap, the $\gamma$ and $\lambda$ terms shape adversarial-case compute.
The risk is gradient masking. STE gradients may systematically underestimate how much attacks on the selector can hurt the model. We therefore add an audit knob: one batch in every $M$ is attacked through the dense path. The setting $M=\infty$ is pure sparse-path training and $M=1$ is fully dense attacks. The empirical question is how large $M$ can get before AutoAttack and adaptive attacks expose a gap.
The cost expectation is modest, and we state it plainly. Per-step cost scales roughly with the fraction of tokens kept, because at DeiT-S resolution the per-token MLP dominates and attention's quadratic term adds only a little. The inner-loop saving should therefore approximate the model's inference speedup. Adaptive Sparse ViT reports about 1.5x throughput on DeiT-S at its clean budget. Tighter budgets should give larger savings, in exchange for accuracy. These savings stack with fast single-step attacks and with the sample-level savings of data pruning for adversarial training, which drops training samples rather than tokens. We treat both as complementary and test one combination.
Baselines
The naive reference is the deployment-relevant "do nothing extra" pipeline. We take the clean Adaptive Sparse ViT recipe on DeiT-S and DeiT-B, with no adversarial training. Its reported clean result is a throughput gain from 770 to 1192 images per second on DeiT-S with a 0.2% accuracy drop (Adaptive Sparse ViT). We expect near-zero robust accuracy, so it anchors the efficiency end.
The strongest competitor is the multi-stage pipeline a careful engineer would build. Stage one is full PGD adversarial fine-tuning of the dense DeiT. Stage two is Adaptive Sparse ViT threshold learning with distillation. Stage three is a final adversarial fine-tune of the pruned model. It is strongest because every stage uses the best available tool with dense-path attacks throughout. It shows what accuracy is achievable when cost is not counted. No reported result was found on Lacuna for robust accuracy of a token-pruned ViT on ImageNet under AutoAttack. The comparison is therefore internal, and we will run it. All arms start from the same pretrained checkpoint, use the same data, ε, and PGD step count, and receive the same hyperparameter search budget, measured in trials. We report total GPU-hours including every stage. Beating it means matching its AutoAttack robust accuracy within one point at equal or lower measured clean-input and attacked-input FLOPs, while using clearly fewer total GPU-hours.
The single-stage dense-attack arm isolates our key ingredient. It uses the same objective and single pass, but with $M=1$, so every attack is computed densely. This is In-Train Pruning's design moved to tokens and to PGD. If sparse-path training matches it, the cost saving is free. If it falls short, the gap measures what pruned-path attacks miss.
The $\gamma=0$ arm with a clean-input budget term isolates the worst-case compute penalty.
A fixed-ratio arm with a static per-layer keep fraction tests the layer-budgeting finding of Piras et al.. It is trained with sparse-path adversarial training at matched average FLOPs.
The parameter-efficient defense is Robustness Tokens, applied to the clean pruned model. It reports that DINOv2-G accuracy under white-box attack recovers from 0.3% to 63.1%, with clean accuracy of 85.2% versus 85.6%, after 200 steps. We will evaluate it with attacker knowledge of the tokens, the setting its "secret key" assumption avoids.
Why not just…?
Why not just adversarially train a dense ViT, or download a robust one, and then prune it? This is the main threat, and it may well win on accuracy. HYDRA shows that naive post-hoc pruning of robust networks collapses at high sparsity. Attacking Compressed Vision Transformers shows that compression raises vulnerability, so the pruned model needs another robust fine-tune, which reintroduces the cost. The multi-stage baseline settles this directly. If it beats us by more than a point of AutoAttack accuracy at matched FLOPs, our contribution reduces to a cost-accuracy trade-off, and we will report it as such.
Why not just use fast single-step adversarial training on the dense model? Single-step training cuts the inner loop from $K$ passes to one, which is a larger saving than ours. However, AutoAttack documents widespread overestimation of robustness for simplified defenses, and In-Train Pruning, which relies on fast training, never faced AutoAttack. The two savings are orthogonal: our method can use single-step attacks through the pruned path. We run that combination, and dense fast training plus post-hoc pruning, under the full evaluation.
Why not just use a fixed keep-ratio and skip the dynamic selector? A static budget cannot be manipulated into spending more compute. Piras et al. found that budget allocation, not clever scoring, drives robust pruning on CNNs. Their result has not been tested on ViTs, and a static budget forfeits the per-image savings Adaptive Sparse ViT exploits. This is a real threat to the dynamic half of the claim but not to the sparse-path training half, which applies equally to fixed ratios. If the fixed-ratio arm matches the dynamic one under attack at equal average FLOPs, we will conclude that dynamic selection is not worth its attack surface in the adversarial regime.
Why not just append Robustness Tokens to a pruned model? They are cheap to train, but they add tokens rather than removing them. Their protection assumes the token values stay secret. Nothing shows the internal outlier activations they restore survive token pruning. Our white-box evaluation of this baseline on the pruned model answers the question.
Proposed Experiments and Decisive Checks
The main experiments use ImageNet-1K with DeiT-S and DeiT-B at $\ell_\infty$ radius 4/255. CIFAR-10 at 8/255, with DeiT-S fine-tuned at 224 resolution, serves for sweeps over $\lambda$, $\gamma$, $M$, and the budget $B$. For each arm we report clean accuracy, AutoAttack robust accuracy, mean and worst-case measured FLOPs on clean and attacked inputs, measured throughput, and total training GPU-hours. As a reference point for token-pruning accuracy at low compute, Training Noise Token Pruning reports 59.93% clean accuracy on DeiT-B at about 5.8 GFLOPs.
Three checks decide the proposal.
The first check is gradient masking. Beyond AutoAttack, we attack every model with three adaptive attacks: PGD through a dense, mask-free surrogate; PGD with the hard top-k selection approximated by a smooth function only in the backward pass; and a rank-distortion attack adapted from CAGE that targets internal ViT selection. Sparse-path training passes only if its worst-case adaptive accuracy stays within one point of the dense-attack single-stage arm.
The second check is compute integrity. We run a pure compute-inflation attack, the $\gamma$-term alone, against every dynamic model. The $\gamma>0$ arm must keep attacked-input FLOPs near $B$, while the $\gamma=0$ arm should drift toward the hard cap.
The third check is the cost curve. We plot AutoAttack accuracy against total GPU-hours for all arms and sweep the audit interval $M$. The proposal succeeds if some $M>1$ lies on the accuracy-cost frontier above both the multi-stage and dense-attack single-stage pipelines.
As a stretch experiment, we will test whether the selector trained with the compute-aware attacker reduces the clean-accuracy loss that Zhang et al. report for robustness-aware selection. Their reference is LLaVA at a 16-token budget, where robust accuracy falls to 42.7% under CAGE versus 49.7% under the baseline attack. We would replace the vision tower's token selector with ours.
References
- Adversarial Robust Model Compression Using In-Train Pruning
- HYDRA: Pruning Adversarially Robust Neural Networks
- Adaptive Sparse ViT: Towards Learnable Adaptive Token Pruning by Fully Exploiting Self-Attention
- Differentiable Joint Pruning and Quantization for Hardware Efficiency
- On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression
- Attacking Compressed Vision Transformers
- Adversarial pruning: A survey and benchmark of pruning methods for adversarial robustness
- Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks
- SAFE: Finding Sparse and Flat Minima to Improve Pruning
- Efficient Adversarial Training With Data Pruning
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- Training Noise Token Pruning