Adversarial training makes image classifiers resistant to small, deliberately crafted pixel changes. It does this by attacking the model at every training step and learning to survive the attack. The attack is an inner loop of several gradient steps, so training a Vision Transformer this way costs several times more than normal training. A recent shortcut is to do the robustness work during masked image modeling, where a model sees only about a quarter of an image's patches and learns by reconstructing the rest. MIMIR attacks the visible patches of a masked autoencoder so as to maximize reconstruction error, and reports 54.28% AutoAttack accuracy with 76.98% clean accuracy for a ViT-Base on ImageNet-1K. We argue that this attack spends much of its effort in the wrong place. Many masked patches cannot be inferred from the visible ones at all, for example a second object that lies entirely under the mask. Inflating the error on those patches is easy for the attacker and teaches the model nothing about robustness.
This is the patch-level version of a well-known flaw in contrastive self-supervised learning, called the semantic consistency assumption: two random crops of one image are assumed to show the same content, even when one crop shows a cat and the other a lamp (Object-Level Representation Learning). Our proposal is to aim the attack only at masked patches that the model can already predict from clean input, and to measure how far the perturbed prediction moves away from the clean prediction, not how far it lands from the ground truth. We expect each attack step to become more informative, so that one or two steps can match the robustness MIMIR gets from a full multi-step attack. The benefit should be largest on cluttered multi-object data, where unpredictable masked patches are most common.
Core Claim
In adversarial masked pretraining, the attacker's objective sums reconstruction error over all masked patches. The gradient of that sum is dominated by patches whose content is not recoverable from the visible set. Those patches have large irreducible error and unstable gradients, so perturbations aimed at them are poor training signal per unit of compute.
We replace the attacker's objective with a predictability-weighted deviation loss. Let $x_v$ be the visible patches, $\delta$ a perturbation with $\|\delta\|_\infty \le \epsilon$, $g$ the masked autoencoder, $\hat{y}_j = g(x_v)_j$ its clean prediction for masked patch $j$, and $e_j$ the clean reconstruction error of that patch, which is already computed in the clean forward pass. The attacker maximizes
$$ \mathcal{L}_{adv}(\delta) = \sum_{j \in M} w_j \,\big\| g(x_v+\delta)_j - \mathrm{sg}(\hat{y}_j) \big\|^2 + \lambda \sum_{i \in V} \big\| g(x_v+\delta)_i - x_i \big\|^2, \qquad w_j = \frac{\exp(-e_j/\tau)}{\sum_{k\in M}\exp(-e_k/\tau)}. $$
Here $M$ and $V$ are the masked and visible index sets, $\mathrm{sg}$ stops gradients, $\tau$ sets how sharply the weights favor predictable patches, and the second term is a visible-patch purification term in the spirit of MAEP, weighted by $\lambda$. The outer model update then minimizes the usual masked reconstruction loss on the perturbed input plus the same deviation term.
The weights add essentially no compute. The testable claim is that at a matched number of attack steps, and therefore matched GPU-hours, this objective gives higher AutoAttack accuracy than MIMIR's objective. We also predict that the advantage grows when pretraining moves from iconic ImageNet images to multi-object COCO scenes.
Research Program
The work has three parts.
Diagnosis. We first check that the premise holds. On a MIMIR-style model we log, per masked patch, the clean reconstruction error and the share of the attack gradient norm that patch contributes. We then test whether gradient mass concentrates on high-error patches.
We also use spatial structure disruption as a probe for why those patches are unpredictable. Following the patch-shuffling protocol of Zhong et al., we shuffle the visible tokens' positions and measure how much each masked patch's error changes. Patches whose error does not depend on layout are predicted from local texture. Patches whose error rises sharply under shuffling depend on global structure. This tells us whether the weights $w_j$ track "inferable from context" or merely "smooth texture". The latter would be a failure mode, since the attack would then retreat to trivially easy background patches.
Method. We build the objective above on a standard masked autoencoder and run a two-by-two ablation. One axis is the attack target: ground truth versus the model's clean prediction. The other is the weighting: uniform versus predictability-weighted. Each cell is run at one, two, and ten attack steps.
Transfer. We carry the best configuration to ViT-Base on ImageNet-1K at a single matched budget. We report the full cost-robustness curve, not a single point.
The spatial-disruption direction appears here as a diagnostic and as a negative control, not as the mechanism. Patch-based negative augmentation and randomization defenses show that scrambling helps against corruptions and naive attacks. The file contains no evidence that scrambling confers worst-case $\ell_\infty$ robustness, and Zhong et al. show it can collapse contrastive feature uniformity. We therefore test "shuffle instead of attack" explicitly, and expect it to fail under AutoAttack.
Baselines
Naive reference points. The first is a plain masked autoencoder, pretrained and fine-tuned identically but with no attack at any stage. This gives the clean-accuracy ceiling and the robustness floor. The second is standard adversarially trained ViT. MIMIR reports 49.7% AutoAttack for it at $\epsilon=4/255$, which we re-run with our code.
Strongest competitor: MIMIR. MIMIR is the same setting with a different attacker objective: all masked patches, ground-truth target, plus an information-bottleneck penalty. Its reported result is 54.28% AutoAttack accuracy and 76.98% natural accuracy for ViT-Base on ImageNet-1K. It also reports AdvXL at 53.0%.
We re-implement MIMIR in the same codebase (easyrobust), with the same data, epochs, mask ratio, fine-tuning recipe and $\epsilon$. We sweep its attack steps over one, two, and ten. We found no reported result on Lacuna for MIMIR's pretraining wall-clock cost or its per-step-count ablation, so we measure both ourselves.
Beating MIMIR means one of two things. Either our method gets higher AutoAttack accuracy at equal attack steps with clean accuracy within one point, or our two-step run reaches MIMIR's ten-step robustness at a fraction of its GPU-hours.
Isolating the key ingredient. The two-by-two ablation is what isolates it. If the gain comes only from the clean-prediction target, or only from the purification term, then the predictability weighting is not the contribution and we will say so.
Single-step training. We include single-step adversarial training stabilized by local-linearity regularization (ELLE), applied at fine-tuning time on a plain masked autoencoder, as the strongest cheap alternative that does not touch pretraining.
Frozen-backbone repair. We include ROB tokens trained on the frozen plain masked autoencoder, as the cheapest post-hoc option. Their reported number is 63.1% accuracy under white-box attack on DINOv2-G. That is a different backbone and attack, so we re-run it on our ViT-Base under AutoAttack.
Why not just…?
Why not just run MIMIR with fewer attack steps? This is the central threat, and the main comparison is built around it. If MIMIR's one- and two-step runs already match its ten-step run, the attack budget is not the bottleneck and our weighting has nothing to fix. We would report that as the finding. Our claim only survives if the matched-step gap is positive and grows with clutter.
Why not just use single-step adversarial training with a stabilizer? Single-step training is prone to catastrophic overfitting, where the model learns to defeat the one-step attacker without becoming robust to multi-step attacks. Stabilizers such as ELLE add back compute. More fundamentally, single-step training works in the supervised min-max regime, where the research file notes a clean-accuracy tax. MIMIR's 76.98% clean accuracy, against the 70–74% range it cites for competing robust models, suggests the pretraining route pays less of that tax. Our experiment runs ELLE at an equal GPU-hour budget, so this is settled empirically rather than argued.
Why not just attach robustness tokens to a frozen masked autoencoder? ROB tokens cost roughly 200 optimization steps and are hard to beat on price. The question is whether a backbone with no robustness in its features can be repaired by a few prepended tokens under AutoAttack, not just under the attack used to train them. We report both. If ROB tokens on a plain masked autoencoder come within two points of our method, the honest conclusion is that robustness belongs post hoc, and pretraining-time attacks are not worth their cost.
Why not just shuffle patches instead of attacking them? Scrambling costs nothing and removes layout shortcuts. However, the known wins are on corruptions (negative augmentation reports +1.7 points on ImageNet-C). Randomization defenses fall to expectation-over-transformation attacks (Xie et al.), which average the attacker's gradient over the random transformations. We include shuffled-visible-token pretraining as a control, and expect it near the plain-autoencoder floor under AutoAttack.
Decisive Experiments
Sweeps. We run ViT-Small on ImageNet-100 (iconic images) and on COCO train2017 (multi-object scenes), pretrained under all arms at one, two, and ten attack steps. Each is fine-tuned with an identical recipe and evaluated with AutoAttack at $\ell_\infty$, $\epsilon=4/255$, clean top-1 accuracy, and measured GPU-hours. COCO-pretrained models are also evaluated on COCO detection, where UniVIP reports 40.8 mAP from COCO pretraining as a non-robust reference.
Headline run. We run one ViT-Base ImageNet-1K pretraining at two attack steps for our method, against MIMIR at two and ten steps.
Pre-specified outcomes. The proposal is supported if three things hold together. Our method beats MIMIR at equal steps. Our two-step run is within one AutoAttack point of MIMIR's ten-step run. The matched-step gap is larger on COCO than on ImageNet-100. A gap that does not grow with clutter would undercut the semantic-consistency explanation even if the method helps.
Guarding against false robustness. Every robust model is checked with IndicatorsOfAttackFailure, and robustness claims are withheld if any indicator fires. This matters because a weighted attacker could create gradient masking, where the model hides its gradients from attackers without actually becoming robust.
Breadth checks. We report ImageNet-C and ImageNet-Patch, since the latter shows $\ell_p$-robust models often gain nothing against patch attacks (ImageNet-Patch). We also report the patch-shuffle drop rate of Zhong et al., to show whether the robust encoder relies on local texture or on global layout.
References
- MIMIR: Masked Image Modeling for Mutual Information-based Adversarial Robustness
- Unsupervised Object-Level Representation Learning from Scene Images
- Adversarial Masked Autoencoder Purifier with Defense Transferability
- Contrastive Learning Relies More on Spatial Inductive Bias Than Supervised Learning: An Empirical Study
- Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
- Mitigating Adversarial Effects Through Randomization
- Robustness Tokens: Towards Adversarial Robustness of Transformers
- UniVIP: A Unified Framework for Self-Supervised Visual Pre-training
- ImageNet-Patch: A dataset for benchmarking machine learning robustness against adversarial patches