Adversarial training teaches an image classifier to resist small, worst-case pixel perturbations. The price is lower accuracy on ordinary, unperturbed images. One cause the literature has identified is that training spends the same perturbation budget on every pixel, so the model must also become insensitive to noise in background regions it never needed. PART showed that shrinking the budget on unimportant pixels recovers 1.2 to 3.18 points of clean CIFAR-10 accuracy without losing robustness. Its importance maps come from GradCAM, a post-hoc heatmap that requires extra backward passes. That roughly doubles training time, and the maps can only be refreshed every 10 epochs.
A separate line of work builds networks whose attention is split into a spatial map (which locations matter) and a channel gate (which features matter). These blocks were designed to be cheap, not robust. We propose to use that spatial map, which such a network computes on every forward pass at no extra cost, as the importance map for allocating the adversarial budget. The same map then does two jobs. During training it decides where the network is attacked, and at test time it decides where the network looks. The central question is whether this coupling between "where we defend" and "where we attend" moves the clean-versus-robust accuracy curve outward, beyond what a separately computed mask achieves.
Core Claim
A convolutional backbone with factorized spatial-channel attention, trained with a per-pixel budget taken from its own clean-input spatial map, should match PART's clean-accuracy gain at roughly the cost of ordinary TRADES training. It should also beat budget masks that come from outside the attention pathway, including one that is free to compute. The second part is the real claim and the one most likely to fail.
By factorized attention we mean attention computed as a spatial map times a channel gate rather than a full joint map. This is the design used in HA-CNN, which splits soft attention into pixel-level "where" and channel-level "what" streams, and in the SEA block, which applies channel attention and then spatial attention in sequence. Both were built for clean accuracy and efficiency; neither was ever attacked.
The claim rests on two prior findings. TORA-ViTs found that robust vision transformers concentrate on object tokens while clean accuracy draws on background context. PART found that defending the background is wasteful. If the model's own attention already separates object from background, then aligning the defended region with the attended region should cost less clean accuracy than an external mask that may disagree with what the network actually uses.
Research Program
The backbone is a WideResNet with a lightweight factorized block after each residual stage. Each block applies a channel gate, in the squeeze-and-excitation style of global pooling followed by per-channel scaling (the same form MSPP used against corruptions). It then applies a single-channel spatial map, in the style of HA-CNN's soft spatial stream. Parameter overhead is small and is matched in a control arm.
Training uses the TRADES objective. TRADES splits robust error into natural error plus "boundary error", the chance that a clean input sits within the perturbation radius of the decision boundary, and it penalizes the second with weight $\lambda$. TRADES already runs a clean forward pass each step. We read the spatial maps from that pass, average them across stages after upsampling to input resolution, and stop gradients through them.
Each pixel $i$ then gets an $\ell_\infty$ budget. It is $\epsilon_i = \epsilon$ if its map value $s_i$ is in the top fraction $q$ of that image's pixels, and $\epsilon_i = \epsilon_{\text{low}}$ otherwise. Here $\epsilon = 8/255$ matches the evaluation radius, and $q$ and $\epsilon_{\text{low}}$ are tuned. The inner PGD attack projects onto this per-pixel box.
The fixed quantile matters. Without it, the network could make its own training easier by marking nothing as important. The stop-gradient removes any direct gradient incentive to shape the map for that purpose. The mask is recomputed at every step, so it tracks the model as it changes, unlike PART's 10-epoch cache.
A second, analytical strand uses the same trained models to address a tension in the file. MSPP found that channel attention raised both clean accuracy and corruption accuracy on CIFAR-10-C. Tsipras et al. and TRADES argue that worst-case robustness must cost clean accuracy. By ablating the channel gate and the spatial map separately under both uniform and self-budgeted training, we measure which attention axis carries any gain. We also test whether MSPP's win-win survives AutoAttack, the standard ensemble of gradient-based and query-based attacks mandated by RobustBench.
Relation to Prior Work
PART supplies the method we reuse: budget reallocation with full $\epsilon$ on important pixels. It also sets the target to beat. We change where the mask comes from and how often it is refreshed, and we test the coupling hypothesis, which PART cannot address because its mask is external to the network.
TORA-ViTs motivates the object-versus-background split. It resolves the trade-off by adding separate clean and robust adapters to a frozen pretrained ViT. We change the training signal of a single network trained from scratch and add no second pathway. TART also reshapes the training perturbation, suppressing perturbations that point off the data manifold, but it needs a pretrained autoencoder and a stored tangent basis. Neither TART nor PART touches the architecture.
The negative constraint comes from Tsipras et al. and TRADES. They show a trade-off must exist, so we claim only a better frontier, not escape from it. We evaluate strictly at uniform $\epsilon$ so that training-time budget cuts cannot masquerade as robustness.
Baselines
The naive reference point is a WideResNet-34-10 trained with uniform-budget TRADES and no attention blocks. A second reference is the same network made slightly wider to match the attention blocks' parameter count. This controls for the chance that any gain is just added capacity. For context, TRADES itself reports configurations reaching 88.64% natural accuracy and 56.61% PGD robust accuracy on CIFAR-10. These are not a single matched AutoAttack pair, so we rerun rather than quote them.
The strongest existing method on this exact question is PART. We run it on our factorized backbone, using GradCAM masks, a 20-epoch burn-in and 10-epoch refresh as the paper prescribes. PART reports a 1.2% to 3.18% natural-accuracy gain over standard adversarial training on CIFAR-10 with "nearly identical" PGD-20 and AutoAttack robustness. No absolute clean or AutoAttack numbers for PART were found on Lacuna, so we reproduce it under our protocol.
A second strong reference is TART. It reports, on CIFAR-10 with WideResNet-32-10, clean accuracy rising from 81.76% to 83.73% and AutoAttack accuracy from 35.89% to 36.58% relative to standard adversarial training. We report our WideResNet-32-10 runs next to these numbers and rerun TART if the autoencoder is reproducible.
Three ablations isolate the key ingredient.
- Architecture only. The factorized backbone trained with uniform TRADES, which shows what the attention blocks give without budget reallocation.
- Random mask. The factorized backbone with a random mask at the same quantile $q$. This checks whether any budget reduction helps.
- Free gradient-saliency mask. The factorized backbone with a mask taken from the input-gradient magnitude of the first PGD step, which costs nothing extra. This is the decoupled control: equally cheap, but not tied to the attention pathway.
All arms share data, augmentation, epochs, the PGD-10 inner attack and the training seeds. Each gets the same budget of about twelve hyperparameter configurations over $\lambda$, $q$ and $\epsilon_{\text{low}}$, selected on a held-out split.
Beating a baseline means one of two things, each holding across three seeds:
- higher clean accuracy at matched AutoAttack accuracy, with the difference larger than the seed standard deviation; or
- a clean-versus-AutoAttack curve, traced over $\lambda$, that dominates the baseline's curve.
Why not just…?
Why not just run PART with GradCAM? If PART on the same backbone matches self-budgeting, the coupling claim fails. The contribution would then shrink to a roughly twofold training speedup (PART reports 2 hours versus 1 hour on an A100) and per-step mask updates. That is useful but modest. The head-to-head arm settles this directly, and we report wall-clock time next to accuracy.
Why not just use the input gradient that PGD already computes as the mask? This is the most serious threat. Tsipras et al. found that robust models' input gradients are perceptually aligned with objects, so this mask may be as good as ours and needs no architectural change. We include it as a first-class arm. If it matches self-budgeting, we will report that PART's benefit needs neither GradCAM nor attention, and the architectural half of the proposal is refuted.
Why not just tune $\lambda$ in TRADES, or use a wider network? TRADES shows that $\lambda$ trades one accuracy for the other along a single curve; it does not move the curve. We trace full $\lambda$ curves for all arms so that improvement means dominating, not relocating. The width-matched arm rules out capacity as the explanation.
Why not just use TORA-style dual adapters? TORA-ViTs requires a frozen pretrained transformer whose attention cost grows quadratically with the number of tokens. At $\lambda = 0.5$ it reports 83.7% clean and 38.0% PGD accuracy on ImageNet. That setting is outside our CIFAR-scale compute, so we treat it as a different regime rather than a direct competitor.
Proposed Experiments
The main study runs on CIFAR-10 and CIFAR-100 with WideResNet-34-10, plus WideResNet-32-10 for comparison with TART. The metrics are:
- clean accuracy;
- AutoAttack $\ell_\infty$ accuracy at $8/255$, the uniform budget;
- mean accuracy over CIFAR-10-C corruptions;
- training wall-clock time.
Because sigmoid gates can hide gradients, we separately report AutoAttack's Square attack, which queries the model without using gradients. Robustness that appears under gradient attacks but disappears under Square would indicate obfuscated gradients, not real robustness.
Three results would decide the claim, with a fourth measurement to catch a subtler failure. First, self-budgeting must beat the random mask; otherwise budget reallocation is not doing the work. Second, it must beat the free gradient-saliency mask at matched robustness; otherwise the coupling hypothesis fails. Third, it must at least match PART in clean accuracy at close to plain TRADES cost.
Fourth, we check whether the network games its own threat model. We measure how much the self-mask's top-$q$ region overlaps GradCAM, and we track this across training. If the overlap falls steadily while training-set robust loss keeps improving, the model may be steering its attention toward regions that are easy to defend rather than regions that matter. In that case, uniform-budget AutoAttack accuracy should expose the problem.
The channel-gate ablation then tells us whether MSPP's gain on both clean and corruption accuracy carries over to worst-case attacks or appears only on natural corruptions.
References
- Improving Accuracy-robustness Trade-off via Pixel Reweighted Adversarial Training
- Harmonious Attention Network for Person Re-Identification
- Coupled Rain Streak and Background Estimation via Separable Element-Wise Attention
- Trade-off between Robustness and Accuracy of Vision Transformers
- Multi-Scale Unrectified Push-Pull with Channel Attention for Enhanced Corruption Robustness
- Theoretically Principled Trade-off between Robustness and Accuracy
- Robustness May Be at Odds with Accuracy
- RobustBench: a standardized adversarial robustness benchmark
- TART: Boosting Clean Accuracy Through Tangent Direction Guided Adversarial Training