Spending the Adversarial Budget Where Overlapping Tiles Disagree

Large images such as street scenes, drone orthomosaics, and pathology slides are too big for a network to process in one pass. Practitioners cut them into overlapping tiles, run the model on each tile, and blend the outputs. That redundancy is usually treated as pure cost, since every pixel in an overlap is predicted two to four times. Adversarial training makes the cost much worse. It hardens a model by repeatedly searching for a small worst-case input perturbation and training on it, and each search needs many extra forward and backward passes. On tiled images, that multiplier is applied on top of the tiling multiplier.

This proposal treats the redundancy as a resource. When several tiles at different offsets predict the same pixel, how much they disagree is an uncertainty estimate the pipeline gets for free. It needs no labels and no dropout sampling. We claim that this cross-tile disagreement, measured after a single cheap gradient step, can tell us which regions of a scene a full multi-step attack will break. Robust fine-tuning can then run the expensive attack only there. Generating each perturbation once on the whole scene, rather than once per tile, removes the overlap multiplier from the attack itself.

Core Claim

Consider tiled dense prediction (segmentation or detection) on high-resolution images, attacked by an adversary who perturbs the whole scene and backpropagates through the blending step. Two changes should let robust fine-tuning match the robustness of "attack every tile every step" at a fraction of the backbone passes. The first is to define the adversarial perturbation $\delta$ on the full scene canvas, so that every tile covering a pixel sees the same perturbed value and the attack gradient is accumulated through the stitcher. The second is to choose which regions receive a full multi-step attack using an offset-disagreement score computed from the overlapping predictions. For a pixel $u$ covered by the set of tiles $T(u)$, with tile $t$ predicting class distribution $p_t(u)$ and blended prediction $\bar p(u)$, the score is

$$D(u) = \frac{1}{|T(u)|}\sum_{t \in T(u)} \mathrm{KL}\big(p_t(u)\,\|\,\bar p(u)\big).$$

$D$ is evaluated on inputs perturbed by one signed-gradient step. Regions are ranked by mean $D$. Only the top fraction receive $K$-step projected gradient descent (PGD, the standard iterative attack), and the rest keep the one-step perturbation.

The intuition is that a perturbation pushing one tile's view across a decision boundary often fails to do the same from a shifted view. High post-step disagreement therefore marks pixels sitting near a boundary that a stronger attack will cross. Because $D$ needs no labels, the same score can select tiles from an unlabeled pool for a consistency-style robustness loss. That is where the active-learning view enters: the network spends compute on what it is unsure about, as PixelPick spends labels on uncertain pixels.

The idea also has to survive a known weakness. Softmax confidence is badly calibrated under adversarial shift, which is why we do not use single-tile entropy. Whether offset disagreement is better calibrated than single-tile confidence is the empirical question this proposal tests.

Relation to Prior Work

The work on reconstructing outputs from overlapping tiles motivates the setting and supplies the stitcher. Distance-decay weighting (Zhou et al., Eggimann et al.), spline windows (Benjdira et al.), and Gaussian center weighting (Edirimuni) all blend redundant predictions to hide seams. All of them treat disagreement between tiles as an error to average away. We instead read it as a signal.

Continuous Patch Stitching shows that seams come from padding and can be removed exactly, by deriving the overlap from the receptive field and using padding-free layers. That result constrains our design. In such a network $D \equiv 0$ by construction, so our signal exists only for off-the-shelf padded CNNs and fixed-patch transformers. Those are the models people actually tile, and they are where we work.

Two papers supply the efficiency logic. SIFT uses an early-layer head to estimate per-sample uncertainty and backpropagates on only half of each batch, giving a 24.5% wall-clock speedup on BERT pretraining. SIFT sifts benign gradient steps. We sift the inner attack loop, and we select spatial regions within an image rather than whole samples. The same paper's warning that greedy selection can plateau above uniform training is a risk we measure directly.

The trainable component comes from Robustness Tokens. It hardens a frozen DINOv2 by tuning ten prefix tokens, raising ADE20K mIoU under attack from 4.7 to 23.3. That paper still generates attacks on fixed-size crops and says nothing about which crops to attack or about stitched scenes. We reuse its parameter-efficient training and replace its attack-generation policy.

Two further papers motivate the threat. Dong et al. report gradient-norm spikes during point-wise adversarial training, which could make cheap one-step probes unreliable. We test for this explicitly. The computational pathology study shows tiled CNNs collapsing under PGD, with AUROC falling from 0.960 to 0.429 on TCGA-RCC, while ViTs degrade less, to 0.827. So hardening tiled models is a real problem in the domains that need tiling.

Research Program

The first stage is diagnostic and fills a gap the file confirms is open: nobody has measured how blending behaves under attack. On Cityscapes and SelvaBox with sliding-window inference at 25% and 50% overlap, we compare three attacks. The first perturbs each tile independently and blends the results, the attack implicitly assumed by per-tile training. The second is a whole-scene PGD attack with gradients through the differentiable blending weights. The third adds expectation over random grid offsets. If blending already averages out per-tile attacks but not scene-level ones, then per-tile adversarial training optimizes the wrong objective, and scene-canvas perturbations are necessary rather than merely cheaper. Alongside this, we record whether $D$ after one step predicts per-region error after full PGD, measured by Spearman correlation and area under the curve for "region flips under PGD-20". We compare it against single-tile entropy, the top-two margin used by PixelPick, MC-dropout variance, and one-step loss increase.

The second stage builds the training method. Each step loads a scene crop large enough to hold several overlapping tiles. It runs one signed-gradient step on the canvas, computes $D$ from the overlapping predictions that tiled inference already produces, picks the top $\rho$ fraction of regions, runs $K$-step PGD on the canvas restricted to those regions, and updates the trainable parameters on the stitched loss. The trainable parameters are ROB-style tokens plus the segmentation head on a frozen DINOv2 backbone. In a secondary setting, they are all weights of a ResNet-based segmenter. For unlabeled scenes, the update is a consistency loss between clean and perturbed blended predictions.

We sweep $\rho$ over 5%, 10%, 25%, 50%, and 100%, and we sweep $K$. Cost is reported as total backbone forward-plus-backward passes and wall-clock time on fixed hardware. Inference stays deterministic: selection happens only during training, so an attacker gains nothing by knowing the selector.

Baselines

The naive reference is the clean model with standard blended tiled inference and no hardening, evaluated under the scene-level attack. It gives the floor, and the hardened methods are judged by how much of the gap to clean accuracy they close.

The strongest existing method is Robustness-Tokens hardening applied per tile, with every tile attacked by full PGD at every step, as in Robustness Tokens. Its reported ADE20K result for DINOv2 is 23.3 mIoU under attack, up from 4.7. We first reproduce this number on ADE20K to validate our attack and training code, then run the same method tiled on Cityscapes and SelvaBox. It is the right competitor because it is already the cheapest strong hardening of a frozen foundation model. If we cannot beat it on cost at equal robustness, the proposal fails.

To make the comparison fair, all methods get the same backbone, the same token count, the same total budget of backbone passes or a stated multiple of it, the same tiling and blending, and the same hyperparameter search of 20 trials each. Everything is evaluated under the same whole-scene attack with expectation over offsets. Beating this baseline means matching its robust mIoU (or RF1₇₅ on SelvaBox) within one point while using at most 40% of its backbone passes, or exceeding it at equal passes.

For full-parameter CNN training on Cityscapes and SelvaBox, we found no reported robust result on Lacuna. We therefore run full-tile PGD adversarial training ourselves as the cost ceiling.

Three ablations isolate the key ingredients. The first keeps scene-canvas perturbations but chooses regions at random at the same $\rho$, which isolates the selector. The second keeps disagreement selection but generates per-tile perturbations, which isolates the canvas sharing. The third replaces $D$ with one-step loss increase, single-tile entropy, or PixelPick-style margin at the same $\rho$, which isolates the choice of uncertainty signal.

Why not just…?

Why not just pick tiles at random with the same budget? Random selection has no scoring cost, and ARMOR shows that random patch injection gives real robustness in aerial detection, holding 0.62 confidence under white-box attack. This is a serious threat. It may succeed if attack-vulnerable regions are spread uniformly across scenes. The random-region ablation settles it. If random matches disagreement selection at $\rho = 10\%$, we drop the selector and report scene-canvas sharing alone as the contribution.

Why not just rank regions by how much the loss rises after one gradient step? This is the most plausible competitor. It is essentially greedy loss-based selection, as in SIFT, and it costs about what $D$ costs. Disagreement has two concrete advantages. It needs no labels, so it also applies to the unlabeled-pool setting where loss cannot be computed. And it should be less sensitive to the gradient spikes that Dong et al. report, because it compares predictions rather than reading a single noisy gradient magnitude. If one-step loss matches $D$ in the labeled setting, we will say so and restrict the claim for $D$ to the label-free setting.

Why not just add random resizing and padding at test time and skip training? Xie et al. recover Inception-v3 from 1.1% to 98.2% under DeepFool at zero training cost. However, random resizing breaks the pixel alignment that overlap blending depends on, and the authors acknowledge vulnerability to attacks that average gradients over the random transformations. We include it under the offset-averaged attack. We expect it to lose clean seam quality, measured as mIoU within a band of pixels around tile borders, and to lose robust accuracy under the averaged attack.

Why not just use a ViT and rely on its built-in robustness? The pathology study's ViT AUROC of 0.827 under PGD is encouraging. However, unhardened DINOv2-G collapses to 0.3% on ImageNet and 4.7 mIoU on ADE20K under strong white-box attack (Robustness Tokens). The naive baseline above measures this directly in the tiled setting.

Proposed Experiments

The primary benchmarks are Cityscapes, with tiled inference over full-resolution frames and mIoU, and SelvaBox (dataset) for crown detection. On SelvaBox, the clean reference is the reported RF1₇₅ of 48.60, and we found no reported adversarial result on Lacuna. ADE20K serves as the sanity check against the reported 23.3 mIoU under attack. TCGA-RCC tiles test transfer to pathology, using a ResNet with full fine-tuning where reported AUROC under PGD falls to 0.429.

All robust numbers use whole-scene PGD-20 with expectation over random grid offsets at two perturbation budgets. We also report a seam-band score, meaning accuracy restricted to pixels within a fixed distance of tile borders. That score shows whether hardening fixes or worsens border fragility.

Three checks decide the project. First, in the diagnostic stage, $D$ after one step must predict PGD-20 failures better than single-tile entropy. Without that, the selector has no basis. Second, the canvas-sharing ablation must show that per-tile attacks overestimate robustness against scene-level attacks, or that canvas sharing cuts attack passes by roughly the overlap factor at equal robustness. Third, the full method at $\rho \le 25\%$ must match full-tile Robustness-Tokens hardening within one point of robust mIoU at no more than 40% of its passes, while beating random and one-step-loss selection at the same budget.

We also track robust accuracy late in training. If the selected-region method plateaus below the full-tile method, as SIFT's analysis warns, we test an annealed $\rho$ that returns to full coverage for the last tenth of training. That fallback is reported as such, not hidden.