Vision transformers are image models that cut a picture into patches ("tokens") and let every token compare itself with every other one. That all-pairs comparison is called global attention. Its cost grows with the square of the token count, so doubling image resolution multiplies attention compute by sixteen. The usual workaround is to coarsen the image, and coarsening erases tiny objects such as distant drones, pedestrians in aerial footage, or small lesions. A growing family of adaptive tokenizers avoids this by spending fine, small patches only where the image looks informative and covering the rest with large coarse patches. Some do this by mimicking the human retina, with a sharp center and a blurry periphery. Others use local image entropy or a learned importance score. Every one of these methods adds a component that ordinary transformers do not have: an allocator that decides where resolution goes. Nobody has asked whether an adversary, or plain sensor noise, can take control of that allocator.
We argue that it can, and that this matters most for exactly the tiny objects these methods are meant to protect. Global attention already has a known weakness: one adversarial patch can capture almost all of the attention weight. On top of that, an adaptive tokenizer can be steered so that its fine tokens land on a decoy while the real tiny target is folded into a coarse token and lost. We call this allocation hijacking. This note proposes to measure it across existing adaptive tokenizers. It then proposes a foveated tokenizer whose allocator has a provable cap on how many fine tokens any small region can claim, paired with a one-line correction to attention for tokens of unequal size. The goal is small-object accuracy at a fraction of uniform high-resolution compute that degrades gracefully under patches and corruptions.
Core Claim
Content-adaptive tokenization adds a second attack surface beyond attention itself. Allocation hijacking can be cheap. For entropy-driven allocators, Gaussian noise from ImageNet-C raises local entropy everywhere and should flatten the allocation signal. A textured, high-entropy sticker is also a natural decoy. Our claim is that two inexpensive design choices recover most of the lost small-object accuracy at matched compute. The first is a per-region cap on fine-token allocation, driven by a coarse global pass rather than by raw pixel statistics. The second is an attention bias for token area. Together they should do this better than global softmax smoothing or adversarially training the allocator.
The prior work divides into three groups by role. The first motivates the problem. Are Vision Transformers Robust to Patch Perturbations? shows that global attention shrugs off natural patch corruption but is hijacked by a single adversarial patch, with a 61.5% fooling rate for DeiT-small against 30.6% for ResNet50. It studies only uniform grids and attributes part of the CNN advantage to a center bias that ViTs lack. Foveation reintroduces something like a center bias, so it is natural to ask whether that helps or hurts.
The second group supplies the mechanism and is the set of methods we attack and compare against. SpiralFovea places concentric rings of shrinking-to-growing patches around entropy hotspots and cuts attention FLOPs by 84% on clean fine-grained benchmarks. APT assigns patch sizes by multi-scale entropy and trims about 30% of tokens on COCO. SG-Former learns a significance map that keeps dense tokens in salient regions. Segment This Thing foveates around an external click and reduces SAM's 4,096 tokens to 172. All four evaluate only on clean images, and the file itself points out that SG-Former's significance map could be manipulated.
The third group constrains the design. Training on Foveated Images Improves Robustness to Adversarial Attacks shows that retina-like blur helps against attacks and corruptions. It does not reduce tokens, though, and it costs clean accuracy unless fixations are chosen well. SAFER-AiD achieves robustness through multi-step reinforcement-learned saccades, eye movements that hop the sharp center from place to place. Those saccades add sequential latency and cannot be differentiated through. Together these papers tell us that foveation can help robustness. They also tell us not to rely on a slow, non-differentiable gaze policy.
Research Program
The tokenizer works in two stages within a single forward pass. A coarse stage tokenizes the full image at a large patch size and runs a few layers of ordinary global attention. This stage is cheap because the token count is small. A light head then scores each coarse cell for "may contain a small object." This plays the role of QueryDet's coarse heatmap (QueryDet), but here it is computed with attention inside a transformer rather than with a convolutional feature pyramid.
The fine stage then allocates a total budget of $B$ fine tokens. The cells with the highest scores receive fine foveal subdivision, but no cell may receive more than $c$ fine tokens. This cap is the key ingredient. Suppose an adversarial or decoy patch overlaps $m$ coarse cells. Then it can capture at most $mc$ fine tokens, and at least $B - mc$ remain for the rest of the image regardless of how the patch is optimized. Uncapped allocators offer no such bound.
Training uses a differentiable top-$k$ relaxation for the allocation step, so the scoring head learns from the detection loss. Finally, fine and coarse tokens are processed together by global attention. Coarse tokens cover more pixels, so we add a log-area term to the attention logits:
$$w_{ij} \propto \exp\!\left(\frac{q_i \cdot k_j}{\sqrt{d}} + \log a_j\right),$$
Here $q_i$ and $k_j$ are the query and key vectors, $d$ is the key dimension, and $a_j$ is the area of token $j$ measured in fine-patch units. The term comes from an exact identity. A coarse token equal to $a_j$ identical fine tokens would receive exactly this total weight under ordinary softmax. The bias therefore makes mixed-size attention consistent with what a uniform grid would compute on smooth regions. This addresses the mismatch between unequal token sizes and models pretrained on regular grids, which SpiralFovea reports as the "raster prior" problem. The term costs nothing. It is an ablation, not the headline.
The analysis half of the program defines an allocation recall metric. It is the fraction of ground-truth tiny-object area that ends up covered by fine tokens. The metric separates failures of where resolution went from failures of attention itself. We then attack each allocator in three ways. The first is an adversarial patch optimized end-to-end through the allocator, in the style of the patch attacks in the robustness paper above. The second is a non-optimized high-entropy texture sticker, which tests black-box decoying. The third is the ImageNet-C corruption suite.
Baselines
The naive reference points are a uniform ViT detector at full resolution and the same ViT at uniform coarse resolution with large patches. The full-resolution model gives the accuracy ceiling at maximal cost. The coarse model gives the cheap floor, where tiny objects should vanish.
The strongest existing adaptive competitors are APT, SG-Former, and SpiralFovea. APT is the most directly comparable because it targets dense prediction and reports roughly 30% fewer input tokens on COCO detection without degradation. SG-Former reports 51.2 box mAP on COCO with its learned token reallocation and represents the learned-allocator alternative. SpiralFovea is the explicitly foveated, entropy-anchored design. Every method is fit into the same ViT backbone and detection head and trained on the same data with the same schedule and tuning budget. Each is set to the same attention FLOPs per image, so that the comparisons differ only in allocation.
For small-object detection, the non-transformer reference is QueryDet. It reports a 2.1 mAP improvement on VisDrone while running 2.3× faster than the high-resolution baseline. HSFANet reports 41.4% AP on VisDrone. We cite both as context, not as matched-compute rivals. Beating the adaptive baselines means two things at equal FLOPs: clean AP_S within one point of the best of them, and a substantially smaller drop in AP_S and allocation recall under patch and noise attacks.
Three ablations isolate the proposal's ingredients. The first removes the per-cell cap but keeps the coarse-pass allocator. The second keeps the cap but scores cells by entropy instead of the coarse pass. The third removes the log-area bias. On the attention side, Smoothed Attention, a softmax temperature above one, is the established hijacking defense. The robustness paper reports that it reduces DeiT-small's adversarial fooling rate from about 60% to 40%. We apply it to every model, alone and combined with our tokenizer.
Why not just…?
Why not just use a fixed central fovea and skip adaptive allocation? A fixed geometry cannot be hijacked, which makes it a genuine robustness baseline. Tiny objects in aerial and drone footage, however, are rarely centered. Segment This Thing gets its accuracy from an external click that tells it where the object is. We include a fixed-center, fixed-budget tokenizer. If it matches our clean AP_S on VisDrone, the adaptive half of the proposal is unnecessary and we will say so.
Why not just adversarially train the allocator? This is the most serious threat. The foveation paper reports that standard adversarial training generalizes poorly to natural corruptions, trailing R-Blur by up to 7 points on ImageNet-C. SAFER-AiD notes the heavy compute that adversarial training requires. Neither result was obtained on allocators, so they do not settle the question. We run patch-adversarial training of the uncapped allocator at the same total compute as our method. The cap's advantage is that it holds for attacks never seen in training. If adversarial training matches us on held-out attacks and corruptions, the cap's contribution reduces to its cost advantage.
Why not just smooth attention? Smoothing addresses attention hijacking, not allocation hijacking. If the target was already folded into a coarse token, no temperature setting restores its pixels. The allocation-recall metric tests this directly. If smoothed models keep high allocation recall under attack, our premise is wrong.
Why not just prune tokens after early layers, as FastV does? FastV still pays full attention cost in the first layers. Its pruning rule is also content-adaptive, so it should be hijackable in the same way. We include it among the attacked allocators.
Proposed Experiments
We train on COCO and VisDrone, and report COCO AP_S as well as overall AP to check that large objects are not sacrificed. We use ARD-MAV as a stress test with extremely tiny targets. For classification-side corruption robustness, we use ImageNet-1K with ImageNet-C. On ARD-MAV, the GLAD reference is 0.80 AP. For ImageNet-C, no reported result found on Lacuna for adaptive tokenizers. For adversarial patches against adaptive tokenizers, no reported result found on Lacuna either. Those numbers are what this study would produce.
Three checks are decisive. First, entropy-driven allocators (APT, SpiralFovea) should lose allocation recall on tiny objects under ImageNet-C Gaussian noise and under the texture sticker, while the coarse-pass allocator should lose much less. If that does not happen, allocation hijacking is not a practical problem. Second, under optimized patches of increasing size, the capped allocator's AP_S should fall roughly in proportion to the number of cells the patch covers, while uncapped allocators should collapse once the patch can attract the whole budget. Third, at matched FLOPs, the full method's clean AP_S on VisDrone should be within a point of the best adaptive baseline. A larger clean-accuracy cost would mean the cap is buying robustness too expensively.
We will also report the converse finding, whichever way it falls: whether a patch landing in the periphery is diluted by coarse tokens, giving the CNN-like center protection that uniform ViTs lack, and whether a patch landing in the fovea is amplified.
References
- Are Vision Transformers Robust to Patch Perturbations?
- SpiralFovea: Input-Adaptive Foveated Tokenization as a Third Lever of Resource-Adaptive Inference
- Accelerating Vision Transformers with Adaptive Patch Sizes
- SG-Former: Self-guided Transformer with Evolving Token Reallocation
- Segment This Thing: Foveated Tokenization for Efficient Point-Prompted Segmentation
- Training on Foveated Images Improves Robustness to Adversarial Attacks
- SAFER-AiD: Saccade-Assisted Foveal-peripheral vision Enhanced Reconstruction for Adversarial Defense
- QueryDet: Cascaded Sparse Query for Accelerating High-Resolution Small Object Detection
- An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models