Vision Transformers can be broken by a single adversarial patch. A patch covering half a percent of an ImageNet image, about one 16×16 token, can pull nearly all of the model's attention onto itself and drive accuracy from about 73% to 0%. Making the model run in low-precision integers does not help: patches built on a full-precision model still fool its 2-bit sibling. Yet every deployed integer Vision Transformer already carries the ingredients of a defense. Its integer arithmetic has finite ranges. The query–key scores that decide where attention goes are stored in bounded registers, and so are the value vectors that carry information between tokens. Today those ranges are set only to preserve clean accuracy.
This proposal sets those ranges deliberately, choosing them per attention head so that no single token can capture more than a fixed share of any head's attention, or inject more than a fixed amount of signal through it. The resulting bound holds for any input, so it holds for any attack. It adds no inference cost and needs no retraining. We then test whether the bound actually stops patch hijacking, whether it blocks patches transferred from full-precision or differently quantized siblings, and how much clean accuracy it costs compared with expensive cross-bitwidth adversarial training.
Core Claim
In scaled dot-product attention, softmax ignores constant shifts. Only the spread of a query's scores across keys matters. Suppose an integer pipeline saturates each head's pre-softmax scores to a window of width $w$. The window is set from clean calibration statistics, so that only scores far outside the clean range are clipped. Then for a sequence of $N$ tokens, any $k$ adversarial tokens can receive at most
$$ \alpha_{\max}(k) = \frac{k}{k + (N-k)\,e^{-w}} $$
of a query's attention. This follows because, at worst, the adversarial scores sit at the top of the window and all others sit at its bottom.
If value vectors are also saturated at magnitude $v_{\max}$, the adversarial tokens' contribution to that query's output is at most $\alpha_{\max}(k)\,v_{\max}$ per head.
The claim has two parts. First, ordinary quantization calibration picks $w$ and $v_{\max}$ far too loose to give a useful bound. We will measure this directly. Second, per-head windows chosen to make the bound tight still preserve clean accuracy to within about 1%. The reason is that clean attention is extremely peaked near zero: FQ-ViT, a post-training method for fully integer Vision Transformers, reports that 98.8% of ViT-L attention values fall below 1/16. That suggests clean score spreads are moderate in most heads.
The new element is the use of integer saturation as an attack-independent cap on attention hijacking, calibrated against that cap. Integer saturation is cheap and is not gradient masking, so the guarantee does not depend on how the attacker optimizes. The bound limits the patch's influence through attention only. It does not limit the patch token's own residual path, so it is not an end-to-end robustness certificate. Whether limiting the attention path is sufficient is exactly what the experiments test.
Where This Sits
Three FP32 lines of work locate the failure precisely. Patch-Fool chooses patches by intermediate attention scores and maximizes the attention they receive. It shows that a single patch drops DeiT-S to 6.25% robust accuracy, and that adversarial training leaves Vision Transformers more fragile than CNNs. Give Me Your Attention (Attention-Fool) skips softmax saturation by attacking pre-softmax dot products directly. It names the structural enablers: large projection singular values and LayerNorm outputs that are not centered on zero. Both papers motivate this proposal, and Attention-Fool also supplies our strongest attack.
Towards Robust Semantic Segmentation via Attention Refinement introduces Max Attention Suppression, a hard cap on attention weights placed before the softmax. It reports roughly a 0.15% clean mIoU cost on ADE20K. That result is the closest mechanism to ours and the main reason to believe the clean-accuracy cost can be small. However, it uses a single global cap, applies only to floating-point models, gives no bound on the value path, and is never tested for transfer.
On the quantization side, FQ-ViT supplies the integer pipeline we build on: Power-of-Two Factor quantization for LayerNorm inputs and Log-Int-Softmax for 4-bit attention maps. It never evaluates attacks. Exploring Patch Attacks in Quantized Networks constrains our expectations. It finds that patches are "quantization-proof": attack success on ResNet-20 falls only from 87% to 74% when an FP32 patch is moved to a 2-bit model. Default quantization therefore buys nothing, and any gain must come from range choice.
TriQDef is the closest joint work. It penalizes edge and HOG similarity of feature maps and input gradients across bitwidths, using curriculum training. It reports 65.8% robust accuracy on ImageNet against 37.1% for patch adversarial training, with a clean accuracy cost within 2%. It never touches attention matrices, and it needs multiple bitwidth passes per batch plus sensitive hyperparameter tuning. Our method needs calibration data only.
On transfer, Quantization Aware Attack shows that multi-bitwidth surrogates strengthen transfer of pixel-level attacks. Publishing Efficient On-device Models shows that a public compressed sibling makes its private server parent far easier to attack. Both supply threat models we reuse, with the attack moved from pixel noise to attention hijacking.
Research Program
The first step is to measure. We will take DeiT-S, DeiT-B and Swin-S, both in FP32 and under FQ-ViT at INT8 and INT4. For each head we will record the pre-softmax score spread and value magnitudes on clean ImageNet calibration images and on Attention-Fool patched images. The diagnostic question is whether hijacking patches drive scores far outside the clean spread, which our clip would cut off, or merely to its upper end, which it would not. Attention-Fool's LayerNorm analysis predicts the former, but this needs checking per head.
The second step is calibration. For each head we choose $w$ and $v_{\max}$ to minimize clean task loss on a few thousand calibration images, subject to a target bound $\alpha_{\max}(1) \le \tau$. Because the constraint is closed-form, this is a one-dimensional search per head. We also add an optional re-centering of LayerNorm outputs, folded into the existing per-channel scaling, to remove the off-origin geometry that Attention-Fool identifies. Sweeping $\tau$ traces a clean-versus-robust curve with no retraining at any point.
The third step is adversarial evaluation, and here gradient masking is the main risk. Clipped regions have zero gradient, which can make a model look robust without being robust. We will attack the integer model in four ways. The first is with straight-through gradients. The second is with stochastic-quantization smoothing in the style of Galloway's work. The third is with a clip-aware variant of Attention-Fool that maximizes the patch's score margin inside the window. The fourth is a gradient-free patch search. The bound itself does not rely on gradients, so the decisive test is the gap between the clip-aware white-box attack and the transfer attacks.
Baselines
The naive reference is FQ-ViT at INT8 and INT4 with its default calibration, attacked by Attention-Fool and Patch-Fool, alongside the FP32 parent. This baseline measures the default ranges and the "quantization-proof" behavior we expect to inherit. The relevant reported number is ViT-T falling from about 73% clean to 0% under a 0.5% Attention-Fool patch, in FP32 (Attention-Fool). Clean accuracy is anchored by FQ-ViT's DeiT-B result of 81.20% against 81.85% for FP32. No reported result was found on Lacuna for Attention-Fool against any quantized Vision Transformer.
The strongest existing defense is TriQDef, retrained on DeiT-B and Swin-S with its published bitwidth curriculum, given a tuning budget equal to its paper's search over $\alpha, \beta, \lambda$, and evaluated with our attacks. Its reported ImageNet robust accuracy is 65.8%, against 37.1% for patch adversarial training. Its own patch setting may differ from ours, so we rerun it rather than quoting that figure as directly comparable.
We also run quantization-aware patch adversarial training (QADT, from the quantized patch study) at matched GPU-hours. The point is to show what training compute buys compared with calibration alone.
The ablations isolate the key ingredient. Max Attention Suppression with a single global cap, applied on top of FQ-ViT, tests whether per-head, bound-driven calibration matters at all. Score clipping without value clipping, and clipping without LayerNorm re-centering, isolate the other two components.
Beating the baselines means the following. Against the clip-aware white-box attack, our method must match or exceed TriQDef's robust accuracy at an equal or smaller clean cost, while using zero training. Separately, it must clearly beat global Max Attention Suppression at the same clean accuracy.
Why not just…?
Why not just add Max Attention Suppression to an FQ-ViT model? This is the most serious threat. If a single global cap performs as well, our contribution shrinks to a reinterpretation of an existing method. We expect it to fall short for three reasons. Clean score spreads vary across heads and layers, so any one cap is either loose in some heads or harmful in others. Max Attention Suppression leaves the value path unbounded. And it was only ever checked in FP32 (RAM). The matched clean-accuracy comparison settles the question. If global capping lands within noise of per-head calibration, we will report that integer saturation alone is the defense and say so plainly.
Why not just do patch adversarial training at low bitwidth? Patch-Fool reports that adversarial training leaves Vision Transformers more vulnerable than CNNs. TriQDef reports that standard patch adversarial training reaches only 37.1% on ImageNet. Adversarial training also costs repeated attack generation in every epoch. Our compute-matched QADT run measures directly whether that cost buys more than calibration does.
Why not just quantize more aggressively and rely on rounding? Patches survive down to 2 bits (quantized patch study). Naive post-training quantization of ViT-L collapses clean accuracy to 3.37% (FQ-ViT). Any apparent gain from lower bitwidth is also exposed to gradient masking. Our INT4-with-default-ranges arm, attacked with smoothed gradients, tests this directly.
Why not just use TriQDef on Vision Transformers? It does run on DeiT-B and Swin-S, and it may well win on robustness. Its cost is multi-bitwidth training passes and hyperparameter tuning, and it acts on edge and texture features rather than on the attention route that hijacking exploits. We also test the two methods combined. If combining them adds robustness, the two methods are complementary rather than competing.
Decisive Checks
Experiments run on ImageNet with DeiT-S, DeiT-B and Swin-S, using single-token and four-token patches. We also run DETR on MS COCO, where Attention-Fool drops mAP from about 53% to under 3%. Metrics are Top-1 accuracy, mAP, robust accuracy, attack success rate, and the measured maximum attention share captured by the patch, which can be checked directly against $\alpha_{\max}$.
The transfer study covers the patch directions FP32→INT8, FP32→INT4, INT4→FP32 and INT8→INT4, all with Attention-Fool. It also includes a multi-bitwidth surrogate in the spirit of Quantization Aware Attack, built from attention losses, and the on-device-to-server setting.
Three outcomes decide the proposal. First, the patch's measured attention share must respect the bound under every attack, including the clip-aware one. If it does not, our model of the integer pipeline is wrong. Second, at $\tau$ settings costing at most 1% clean accuracy, robust accuracy against the clip-aware white-box attack must rise substantially above default FQ-ViT. If the gain appears only under straight-through gradients, it is gradient masking. Third, transfer success from FP32 and from sibling bitwidths must fall below white-box success on the calibrated model.
If the bound holds but accuracy under attack stays low, the patch is acting through its own residual stream rather than through attention. That would also be a useful negative result: it would show where hijacking actually lives in quantized Vision Transformers.
References
- FQ-ViT: Fully Quantized Vision Transformer without Retraining
- Patch-Fool: Are Vision Transformers Always Robust Against Adversarial Perturbations?
- Give Me Your Attention: Dot-Product Attention Considered Harmful for Adversarial Patch Robustness
- Towards Robust Semantic Segmentation against Patch-Based Attack via Attention Refinement
- Exploring the Robustness and Transferability of Patch-Based Adversarial Attacks in Quantized Neural Networks
- TriQDef: Disrupting Semantic and Gradient Alignment to Prevent Adversarial Patch Transferability in Quantized Neural Networks
- Quantization Aware Attack: Enhancing Transferable Adversarial Attacks by Model Quantization
- Publishing Efficient On-device Models Increases Adversarial Vulnerability