Masked Latent Alignment for Frozen Vision Transformers That Stay Accurate and Explainable Under Attack

Large pretrained vision models are both easy to fool and expensive to explain. A tiny, carefully crafted change to an image (an adversarial perturbation) can flip the prediction. Producing a heatmap of which image regions drove a decision typically takes hundreds of extra model evaluations. The usual fix for fragility is adversarial training, which retrains every weight on attacked images generated inside the training loop. It is too costly for billion-parameter models, and it tends to lower accuracy on ordinary images. A cheaper route is now available. Robustness Tokens freezes the whole model and learns only ten extra input vectors, called tokens, which pull the model's internal features for attacked images back toward the features of the clean images. A parallel line of work amortizes explanations: it trains a small side network once, so each explanation later costs a single forward pass. Nobody has asked whether these two cheap fixes are compatible. When a token-defended model is attacked, does its fast explanation still describe what the model is actually doing?

We propose one training signal that serves both goals. We still learn a few input tokens for a frozen transformer, but we align features under random patch masking. The model's internal features for a perturbed image with a random subset of patches hidden should match its features for the same masked clean image. Fast mask-based explainers are trained exactly on how features change when patches are hidden. If those masked features no longer move under attack, then the explainer's training targets no longer move either. Masked views are also shorter token sequences, so generating attacks on them is cheaper than on full images. The work is new in two ways. It tests whether faithfulness of real-time explanations under attack can be obtained as a byproduct of latent alignment. It also tests whether masking reduces the attack-generation cost that even token defenses still pay.

Core Claim

Let $h_0(x)$ be the frozen backbone's patch-level features for image $x$, and $h_\tau(x)$ the features when learned tokens $\tau$ are appended. Let $M$ be a random binary patch mask (the visible patches) and $\delta$ a bounded perturbation. We train only $\tau$ with

$$ \mathcal{L}(\tau)=\mathbb{E}_{x,M}\Big[\,\|h_\tau(M\odot x)-h_0(M\odot x)\|^2 \;+\; \lambda\,\|h_\tau(M\odot(x+\delta^\ast))-h_0(M\odot x)\|^2\Big], $$

The first term keeps clean behavior unchanged, and the second corrects attacked behavior. Here $\delta^\ast$ is found by $k$ gradient steps that maximize the second term on the masked view, and $\lambda$ weights correction against invariance. The full, unmasked image is included as the case where $M$ is all ones.

We claim three things. First, robust accuracy should match full-image robustness tokens at lower attack-generation cost, since per-step cost scales with the number of visible patches. Second, a meta-attribution explainer refit to the tokenized model will remain faithful on attacked inputs, while one paired with ordinary full-image tokens will not. Third, clean accuracy will stay within the small gap robustness tokens already show. The second claim is the one that separates this from existing work, and it can fail. Full-image alignment may already restore masked-view behavior, and we treat that outcome as a finding rather than a nuisance.

Where This Sits

The method comes from two sources. Robustness Tokens supplies the frozen-backbone, token-only alignment. On DINOv2-Giant it kept clean ImageNet accuracy at 85.2% (baseline 85.6%) and raised white-box accuracy under attack from 0.3% to 63.1%, lifting clean-to-attacked feature cosine similarity from 0.12 to 0.89. That paper never studies explanations, and it aligns only full images. TVE supplies the explainer. It learns how input patches contribute to backbone embeddings from masked views, then turns these "meta-attributions" into heatmaps for any linear head in one pass. It runs at about 100 images per second against fewer than 10 for LIME or RISE. CoRTX is a related amortized explainer, reaching 0.00015 s per instance with 5% of the ground-truth labels. Neither TVE nor CoRTX was evaluated on adversarial inputs.

Our key observation concerns TVE's design. TVE's explainer is a separate network reading the raw image, and it is trained against one fixed backbone. Under attack, it can keep producing a clean-looking map while the backbone's prediction flips. In that case the explanation describes the clean image rather than the model's decision.

Three papers explain why this matters. Proper Network Interpretability Helps Adversarial Robustness and its companion Visual Interpretability Alone Helps Adversarial Robustness introduce "interpretability sneaking," where a prediction changes but its explanation does not. They show that measuring explanation change for both the true class and the attacker's class exposes it. They also show explanation-consistency training bounds error under attack. Their method, however, uses CAM heatmaps, which require a global-average-pooling CNN, and it keeps a full inner attack loop with a 2–10% clean-accuracy cost. We borrow their two-class discrepancy as a metric, not their training recipe.

Bridging Adversarial Robustness and Gradient Interpretability explains why full adversarial training produces clean gradient heatmaps. It also shows the price: CIFAR-10 accuracy fell from about 74% to 60%. That result constrains us, because it indicates that retraining weights for invariance is what costs accuracy. We therefore never touch backbone weights.

Adversarial Examples Are Not Easily Detected constrains our evaluation. An attacker aware of a defense can match intermediate features, so every claim must hold against adaptive attacks that know the tokens and target the explainer jointly.

Research Program

We build three things. The first is masked robustness tokens for DINOv2 and DeiT-III backbones, trained with the objective above. Masks keep 25%, 50%, or 100% of patches, and hidden patches are dropped from the sequence rather than zeroed, so attack steps on masked views are cheaper. The second is a TVE explainer refit to each tokenized backbone using clean images only. The explainer is never shown attacks, so it cannot learn to ignore them. Any faithfulness under attack must therefore come from the tokens changing the backbone's behavior.

The third is an evaluation harness that measures four things. The first is robust accuracy. The second is explanation faithfulness, scored by insertion and deletion curves computed on the attacked image against the defended model's own outputs. The third is explanation stability, measured by Kendall's tau between clean and attacked maps. The fourth is a sneaking rate: the fraction of successful attacks whose two-class explanation discrepancy stays below the level seen between two clean augmentations of the same image. A defense that raises stability while also raising sneaking is hiding failures, and we will report it as a failure.

Baselines

The naive reference is the frozen backbone with a TVE explainer and no defense. It fixes the floor for robust accuracy, and it shows how badly fast explanations degrade under attack. For this setting there is no reported result found on Lacuna for TVE faithfulness under adversarial perturbation, so we measure it ourselves.

The strongest existing method is full-image robustness tokens from Robustness Tokens. Its reported figures are 63.1% white-box accuracy and 85.2% clean accuracy on ImageNet-1k with DINOv2-Giant. It is the right competitor because it already solves robustness cheaply with the same frozen-backbone constraint. We re-run it with our code on the same backbones, data, token count, optimizer, and step budget. The file does not give its attack settings, so we match attack strength across methods rather than copy its protocol. It is beaten only if masked tokens match its robust accuracy within noise at fewer total backbone FLOPs, and also yield higher attacked-image insertion/deletion scores and a lower sneaking rate.

The central ablation isolates the ingredient that matters: full-image tokens given the same total compute as the masked variant, split into more steps. If this closes the explanation gap, masking adds nothing beyond cheaper attacks.

As expensive references we include full-weight adversarial fine-tuning of a smaller ViT under a fixed GPU budget. On CIFAR-10 we also cite the diffusion-data result of Better Diffusion Models Further Improve Adversarial Training: 70.69% AutoAttack robust and 93.25% clean accuracy with a WRN-70-16 trained on 50M synthetic images. That result is a compute ceiling for context, not a like-for-like competitor. For explanation stability, the reference point is the Kendall's tau of 0.913 (versus 0.278 for IG-Norm on MNIST) from Proper Network Interpretability Helps Adversarial Robustness. It was obtained on CNNs with CAM, and we report our tau on the same scale while noting the architectural mismatch.

Why not just…?

Why not just use ordinary robustness tokens with off-the-shelf TVE? This is the real threat. Full-image alignment already pushes cosine similarity to 0.89, and that may carry over to masked views. The ablation settles it. If full-image tokens give equal attacked-image faithfulness and sneaking rate at matched compute, then our explanation claim is false. The contribution would then shrink to a cheaper-attack result plus the first measurement of fast-explainer behavior under attack, and we would report exactly that.

Why not just adversarially train the explainer? An explainer made insensitive to perturbations will output the clean image's map even when the model's prediction has flipped. That is precisely the sneaking failure Visual Interpretability Alone Helps Adversarial Robustness warns about. We include this variant and expect high stability together with a high sneaking rate. Low faithfulness against the defended model's actual outputs on attacked inputs would confirm that it hides attacks rather than explaining them.

Why not just read gradient heatmaps off an adversarially trained model? Those maps are interpretable. However, Bridging Adversarial Robustness and Gradient Interpretability reports a CIFAR-10 accuracy drop from about 74% to 60%, and each map still needs a backward pass. That rules the approach out for real-time use on frozen foundation models. The full-weight adversarial fine-tuning baseline measures both costs directly in our setting.

Why not just drop the attacks and train tokens with Guided Complement Entropy? Guided Complement Entropy preserves clean accuracy with no attack loop, but it retained only about 15% accuracy under momentum iterative attacks on CIFAR-10. We run a token-only version as the zero-attack-cost point on the compute curve. We expect it to fall well short, and if it does not, the whole attack loop is unnecessary.

Proposed Experiments and Decisive Checks

The main experiment covers ImageNet-1k with DINOv2 (Base and Giant) and DeiT-III backbones. Transfer checks use Imagenette and CIFAR-10 linear probes, which match TVE's evaluation sets. We sweep mask ratio (25%, 50%, 100%) and attack steps $k$ (1, 3, 10). We report clean accuracy, robust accuracy under $\ell_\infty$ attacks at $\epsilon=8/255$, total backbone FLOPs and GPU-hours, attacked-image insertion/deletion, Kendall's tau, and sneaking rate. Attack sanity is checked with IndicatorsOfAttackFailure, so that robustness gains are not artifacts of weak optimization.

Adaptive evaluation is mandatory. Attackers know the tokens, which addresses the secrecy weakness noted for robustness tokens. They optimize three objectives: prediction flip alone, explanation corruption alone, and a joint objective that flips the prediction while matching the clean explanation.

Three results decide the proposal. The first is whether full-image tokens leave a measurable faithfulness gap on attacked images that masked tokens close. The second is whether masked tokens reach comparable robust accuracy at lower FLOPs, rather than trading robustness for speed. The third is whether the joint adaptive attack drives sneaking back up. If the first check fails, the explanation benefit comes for free from existing tokens. If the third fails, the defense only moves the problem. A positive result on all three would give a single cheap, weight-free procedure for frozen transformers that keeps clean accuracy within about half a point, restores predictions under attack, and keeps 100-image-per-second explanations honest about what the model does.