Most vision transformers cut an image into a fixed grid of square patches, so a single token often mixes part of an object with part of the background. A newer family of "content-aware tokenizers" draws token boundaries along object edges instead, so that each token covers one coherent piece of the scene. A separate body of work shows that detectors can be fooled without touching the target at all: a perturbation placed in the background can make a stop sign vanish or make a ghost object appear. These two lines of work have never been put together, and the combination raises a sharp question. Clean object-aligned tokens should keep background noise out of the object's tokens, which would make contextual attacks harder. But the tokenizer that decides where the boundaries go now looks at the image too, often globally. An attacker could therefore perturb the background to move the boundaries, merging the object into the background or starving it of tokens before any attention layer runs.
This note proposes a measurement that splits a contextual attack's damage into two parts: damage that flows through the boundary decision and damage that flows through the token features. It also proposes a design rule that follows from that split. The rule is that the boundaries inside an object's region should depend only on pixels within a fixed radius of that region. This turns "the attack cannot move this object's boundaries" from a hope into a property that holds by construction. Our central prediction is that existing adaptive tokenizers whose boundary or budget decisions are global are more exposed to out-of-box attacks than rigid grids, almost entirely through the boundary channel. Making the partition local should remove that channel at small clean-accuracy cost and leave a feature-channel residue no worse than a grid model's.
Core Claim and the Decomposition
Write a tokenized model as $f(x, P)$, where $x$ is the pixel input and $P = \pi(x)$ is the partition (the assignment of pixels to tokens) produced by the tokenizer $\pi$. Token features are pooled from $x$ within the regions of $P$, so $f$ is well defined for any pairing of pixels and partition. For a clean image $x$, an attacked image $x'$ and a task loss $L$ such as detection loss on the target, the attack's effect splits as
$$\Delta_{\text{part}} = L(f(x, \pi(x'))) - L(f(x, \pi(x))), \qquad \Delta_{\text{feat}} = L(f(x', \pi(x))) - L(f(x, \pi(x))).$$
$\Delta_{\text{part}}$ is the harm from adversarial boundaries applied to clean pixels. $\Delta_{\text{feat}}$ is the harm from adversarial pixels pooled into clean boundaries. A residual interaction term, $L(f(x',\pi(x'))) - L(f(x,\pi(x))) - \Delta_{\text{part}} - \Delta_{\text{feat}}$, is also reported. We call this the partition-swap diagnostic. Rigid grids have $\Delta_{\text{part}} = 0$ by definition, so the diagnostic says exactly what adaptivity adds or removes.
The design rule is radius-bounded partitioning. Let $B$ be any image region and $B_r$ its dilation by $r$ pixels. A partitioner is $r$-local if the restriction of $\pi(x)$ to $B$ is a function of $x$ restricted to $B_r$ only. For an out-of-box threat model whose perturbation lies at distance greater than $r$ from the target box, an $r$-local tokenizer guarantees $\Delta_{\text{part}} = 0$ on the target, with no certification procedure and no extra inference passes. Existing tokenizers are not $r$-local. Differentiable Hierarchical Visual Tokenization (∂HT) merges superpixels hierarchically and picks the number of tokens with a global information criterion. CAT chooses a whole-image compression ratio from an LLM's reading of a caption. Region-Native Visual Tokenization assigns pixels to regions through learnable queries attending globally. Subobject-level Image Tokenization (EPOC) predicts boundaries with a SegFormer whose receptive field spans the image, then floods with watershed. Each of these decisions is a channel through which a distant perturbation could reach the target's tokens.
Relation to Prior Work
Two kinds of prior work motivate the question. The first established that contextual attacks are real. Role of Spatial Context in Adversarial Robustness for Object Detection shows that a non-overlapping corner patch blinds a convolutional detector. Out-of-Bounding-Box Triggers forces intermediate features to mimic an empty scene. Structured Universal Adversarial Attacks on Object Detection for Video Sequences confines perturbations to backgrounds. All three target CNN detectors with fixed spatial structure, so none of them has a boundary channel to exploit. The second showed that adaptive tokens help on clean data. This includes the tokenizers above and SPFormer, which uses soft superpixel cross-attention and was evaluated only against benign rotation and occlusion. None of these papers measures adversarial behavior.
Two papers sit closest to our proposal. Robust Superpixel-Guided Attentional Adversarial Attack uses superpixels to shape the attacker's noise against ordinary CNN classifiers. We instead study models whose tokens are superpixels, and we reuse its superpixel-constant noise as one attack family. Understanding and Defending Patched-based Adversarial Attacks for Vision Transformer (ARMRO) explains how a patch token hijacks attention on grid ViTs and masks anomalous tokens. It studies on-object patches on rigid grids, whereas we study off-object perturbations on adaptive partitions.
One constraint shapes the threat model. Benchmarking and deeper analysis of adversarial patch attack on object detectors reports that purely contextual perturbations are fragile to geometric change. For that reason we evaluate digitally first and treat transformation robustness as a secondary axis, not a headline.
Research Program
The first step builds a common detection testbed. A plain ViT-B/16 backbone with a simple detection head is trained on PASCAL VOC 2007 and on MS COCO. The patch embedding is then swapped for ∂HT's retrofit tokenizer, EPOC's partitioner, and SPFormer's superpixel branch, all on the same backbone weights, schedule, and data. A second, smaller track runs ADE20K segmentation, where ∂HT already reports clean numbers.
The second step designs attacks that go after the boundaries directly. Every perturbation is constrained to pixels farther than $d$ pixels from the target box. We use two threat models: a universal corner patch under 6% of image area, following Saha et al., and an $\ell_\infty$-bounded background perturbation. Beyond the standard detection-suppression loss and the empty-scene feature-guidance loss of Lin et al., we add two losses aimed at the partition. A merge loss rewards making object pixels share tokens with background pixels. A starvation loss rewards reducing the number of tokens allocated inside the target box. The tokenizers use discrete steps such as argmax, watershed, and merge decisions, so gradients are obtained with the straight-through and mean-injection relaxations those papers themselves use for training. These are supplemented by a backward-pass-differentiable approximation and a gradient-free search over boundary-map perturbations, so that a tokenizer cannot look robust merely because its gradients are broken.
The third step runs the partition-swap diagnostic on every attacked image, for every tokenizer and every attack.
The fourth step builds and tests the $r$-local variants. For the EPOC-style partitioner, the boundary network is replaced by one with a bounded receptive field of radius $r$, made of stacked small convolutions with no global pooling, and the watershed is seeded from local maxima only. For ∂HT, the global count selection is replaced by the same criterion applied independently per tile of side comparable to $r$, with merges allowed only within dilated tiles. The partition is then provably unchanged inside the target box for $d > r$. We sweep $r$ against clean mAP to trace the cost of locality.
Baselines
The naive reference point is the rigid-grid ViT-B/16 detector, undefended, attacked with the same out-of-box budget. It fixes $\Delta_{\text{part}}=0$ and gives the feature-channel vulnerability of standard tokenization. If an adaptive tokenizer is not worse than this under attack, the "new attack surface" hypothesis fails. There is no reported result on Lacuna for out-of-box attacks against ViT detectors. The closest reported reference is a CNN: YOLOv2 dropping from 76.85% to 56.24% mAP on VOC 2007 under a sub-6% corner patch (Role of Spatial Context). We use that drop as a sanity check on attack strength, not as a direct comparison.
The second baseline is the unmodified adaptive tokenizers: ∂HT, EPOC, and SPFormer, each retrofitted onto the same backbone. ∂HT is the strongest clean competitor in the file, reporting 83.2% versus 82.6% ImageNet top-1 for ViT-B/16 and 53.2 mIoU on ADE20K (Differentiable Hierarchical Visual Tokenization). The local variants must stay within about one point of its clean accuracy while cutting its attacked-mAP loss. Each receives the same training schedule and the same attack budget, including the boundary-targeted losses so the comparison is adaptive.
The strongest existing defense against exactly this threat is Grad-defense (Role of Spatial Context). It penalizes gradient attribution outside ground-truth boxes, keeps 76.09% clean mAP on VOC 2007, and recovers about 10 mAP under attack on YOLOv2. We apply it to the grid ViT detector with a matched training compute budget, counting its double-backpropagation cost against it. Its regularization weight is tuned with the same number of trials as our $r$. Beating it means higher attacked mAP at equal clean mAP and equal or lower training cost. Grad-defense combined with $r$-local tokenization is also tested, since the two may stack.
The key ablation separates locality in the tokenizer from locality in attention. It compares four models: an $r$-local tokenizer with global attention, a global tokenizer with windowed attention, both, and neither. A further ablation uses classical SLIC, which is already approximately local because each cluster searches a bounded window. This tells us whether learned local boundaries are needed at all.
Why not just…?
Why not just regularize the receptive field with Grad-defense on a grid model? It is the most direct competitor and may well close most of the feature channel. However, it needs box annotations and second-order gradients, and Saha et al. report it still leaves a substantial gap below clean mAP. More importantly, it says nothing about adaptive tokenizers, whose boundary decisions sit upstream of any detection-loss gradient it regularizes. If grid-plus-Grad-defense matches the best adaptive model under attack, then for robustness purposes adaptivity buys nothing, and we would report that plainly. The stacking experiment shows whether the two mechanisms address different channels.
Why not just restrict attention to local windows instead of touching the tokenizer? Windowed attention limits how far features travel. If the partition is global, though, a distant perturbation can still change which pixels the target's tokens contain. The 2×2 ablation measures this directly. If windowed attention alone drives $\Delta_{\text{part}}$ to near zero, which can happen when the tokenizer's boundaries are mostly set by local evidence anyway, then the design rule is unnecessary and the contribution reduces to the diagnostic.
Why not just use a fixed classical superpixel method like SLIC, which is local and has no learned gradients? This is a real threat to the method half of the proposal. SLIC is roughly $r$-local by construction. Its non-learned boundaries are not immune to approximated-gradient attacks, but locality bounds their reach. The cost is that SLIC boundaries are less object-aligned, which earlier adaptive-tokenizer work implies costs clean accuracy. If SLIC tokens match the learned local tokenizer on both clean and attacked mAP, we will recommend SLIC and present the learned local variant only as a negative result.
Why not just detect and mask the adversarial tokens at test time, as ARMRO does? ARMRO flags tokens with anomalous attention columns. Off-object attacks that work by erasing the target's features, or by moving boundaries, need not create such spikes. ARMRO also requires two inference passes, and its own authors report robustness falling to about 55% under adaptive attacks (ARMRO). We run it on the grid model under our threat model to check whether it catches out-of-box perturbations at all.
Decisive Checks
The first decisive check is the partition-swap table on VOC 2007 and COCO. The hypothesis is supported if, for ∂HT and EPOC under merge and starvation attacks, $\Delta_{\text{part}}$ accounts for most of the attacked-mAP loss and their total loss exceeds the grid baseline's. It is refuted if $\Delta_{\text{part}}$ stays small under adaptive boundary attacks. That outcome would itself be useful: it would mean adaptive tokenizers are a free robustness gain against contextual attacks, and Gap 2's "intrinsic defense" reading would be correct.
The second check is the locality sweep. Measured $\Delta_{\text{part}}$ on the target must be exactly zero for $d>r$, which verifies the construction. Attacked mAP should then rise toward or above the Grad-defense grid model while clean mAP stays within a point of unmodified ∂HT.
The third check compares the feature channel $\Delta_{\text{feat}}$ between local adaptive tokens and grid tokens at matched token counts. This answers whether object-aligned tokens, once their boundaries are protected, actually resist feature leakage better than squares.
Finally, all attacks are repeated under random scale and affine jitter. This checks whether boundary attacks inherit the geometric fragility reported for contextual patches, or, in line with the superpixel-attack results of Dong et al., survive it better.
References
- Differentiable Hierarchical Visual Tokenization
- CAT: Content-Adaptive Image Tokenization
- Region-Native Visual Tokenization
- Subobject-level Image Tokenization
- Role of Spatial Context in Adversarial Robustness for Object Detection
- Out-of-Bounding-Box Triggers: A Stealthy Approach to Cheat Object Detectors
- Structured Universal Adversarial Attacks on Object Detection for Video Sequences
- SPFormer: Enhancing Vision Transformer with Superpixel Representation
- Robust Superpixel-Guided Attentional Adversarial Attack
- Understanding and Defending Patched-based Adversarial Attacks for Vision Transformer
- Benchmarking and deeper analysis of adversarial patch attack on object detectors