Where Sliding-Window Attention Learns Where the Image Ends

Convolutional networks are supposed to treat an object the same way wherever it appears. In practice they learn the absolute position of every pixel, because the zeros padded around the image border act as a landmark the network can count its distance from. Sliding-window attention models were partly motivated as a fix. The best known is the Neighborhood Attention Transformer (NAT), where each pixel attends only to its $k \times k$ nearest neighbors. When that window would run off the image, NAT slides it inward, so border pixels still see $k^2$ real neighbors instead of padding. Nobody has checked whether this trick trades one landmark for another. A pixel at the edge now sits at the corner of its own window rather than the center. Its learned relative-position bias is indexed by lopsided offsets, so the attention it computes differs systematically from that of an interior pixel. That is exactly the kind of asymmetry that, in convolutions, lets networks triangulate their coordinates.

This proposal first measures how much absolute position modern windowed-attention backbones leak, and through which channel. It then tests one specific redesign. The redesign handles the border with masked, entropy-corrected truncated windows, and it adds a single explicit, switchable coordinate input so that tasks needing a position prior still get one. The bet is that accidental boundary leakage and useful position information can be separated. If they can, shift consistency would improve on classification without the collapse in saliency and segmentation performance that removing padding causes in CNNs.

Core Claim

In sliding-window vision transformers, the rule for handling image borders is a major and unmeasured source of absolute-position information. The two main candidate sources are NAT-style inward re-centering combined with relative position bias, and the zero padding inside convolutional patch-embedding and downsampling layers. We claim that if this leakage is routed through an explicit, ablatable coordinate channel instead, two things become possible together. Classification can be more consistent under natural, non-circular shifts. Position-dependent tasks can keep the performance that accidental leakage currently supplies.

Prior work fixes one of these properties and gives up the other. Islam et al. remove zero padding from a CNN and watch PASCAL-S saliency F-measure fall from 0.66 to 0.32. Kayhan & van Gemert remove leakage with full convolution and gain shift robustness, but give up the prior. Our test is whether the explicit channel recovers what the first lost while keeping what the second gained.

What Is Known and What Is Not

The mechanism of leakage is well characterized in CNNs, and those studies motivate this work. PosENet (Islam et al.) attaches a small readout to a frozen backbone and asks it to regress coordinate ramps. Even on pure noise input, ResNet152 reaches Spearman correlations of 0.93 to 0.99, so the position signal can only come from the border. Garcia-Gasulla et al. use a Kolmogorov–Smirnov test on border-versus-center activations and find that 1.5–3% of filters are dedicated "padding-aware neurons." Alsallakh et al. show that asymmetric padding produces lopsided learned kernels and detection blind spots. Lee et al. show that anti-aliasing removes local jitter, yet features still decorrelate globally as an object crosses the frame, and they attribute the residual to boundary encoding. We reuse all four diagnostics as instruments.

Some work constrains the design. Reflection padding is not neutral. It still leaks coordinates (Islam et al. 2021, Zhu et al.), and it distorts geometry at the border (Innamorati et al.). Circular padding gives the clean guarantees of Chaman & Dokmanic and of the alias-free ViT of Michaeli & Soudry, but it glues unrelated image edges together, which Li et al. show harms restoration. Explicit boundary kernels (Innamorati et al.) shield interior pixels, but they do so by hard-wiring position into the weights. That is the opposite of our goal, so we treat them as a contrast rather than a component.

Our test bed comes from the attention side. Hassani et al. introduce NAT and its re-centering rule, and DiNAT extends it with dilated neighborhoods. Neither quantifies leakage. TEAFormer (Enhancing Image Restoration Transformer via Adaptive Translation Equivariance) preserves translation equivariance in restoration transformers and openly lists boundary processing as unresolved. SI-Score (Yung et al.) finds global ViTs smoother near edges than CNNs, but it never tests windowed ones.

Research Program

The first stage is a factorial measurement study on a small NAT, crossing three factors. The first factor is the attention border rule. It has four settings: re-centering as in NAT, zero-padded keys, reflection-padded token maps, and masked truncation, where the border query simply attends to its fewer valid neighbors. The second factor is the padding in the convolutional stem and downsamplers. It has four settings: zero, reflect, full convolution as in Kayhan & van Gemert, and circular. The third factor is whether relative position bias is on or off. Every cell gets four measurements:

  • PosENet Spearman correlation on noise inputs.
  • The fraction of attention heads and channels flagged as border-aware by the KS diagnostic, adapted to heads by testing per-head attention entropy and output distributions.
  • Zero-input foveation maps.
  • SI-Score location heatmaps.

This stage alone answers Gap 1 of the literature. It is publishable even if the redesign fails.

The second stage is the redesign, which has two parts. The first part is masked truncation: border queries attend only to valid keys. That avoids both the zeros that dilute attention and the lopsided offsets introduced by re-centering. Masked truncation creates its own giveaway, however. A query with $n < k^2$ keys has systematically sharper or flatter attention, which is the "attention collapse" signature in another form. We correct this by scaling the logits so that expected attention entropy does not depend on key count:

$$ a_{ij} = \mathrm{softmax}_j\!\left(\frac{\log k^2}{\log n_i}\cdot\frac{q_i^\top k_j}{\sqrt{d}}\right), $$

Here $n_i$ is the number of valid keys for query $i$, $k^2$ is the full window size, and $d$ is the head dimension. The ratio comes from requiring the softmax over $n_i$ near-uniform logits to have the same entropy scale as over $k^2$. The relative position bias is kept, but only entries for offsets that exist are used, so no asymmetric offset table is learned for borders.

The second part is the explicit coordinate channel. A single learned gate adds Fourier features of normalized absolute coordinates at the first stage. It is trained with the gate active, and at test time it can be zeroed to measure how much the model relies on it. The gate is the only sanctioned route for absolute position.

Baselines

The naive reference is NAT-Tiny exactly as released. It uses re-centering, relative position bias, and zero-padded convolutional downsamplers, and it is trained with the published recipe. Hassani et al. report 83.2% ImageNet top-1 for NAT-Tiny, 48.4% ADE20K mIoU, and 51.4% COCO mAP. No reported result was found on Lacuna for NAT's shift consistency or PosENet correlation, so we measure both ourselves. Beating this baseline means matching its top-1 accuracy within 0.3 points while raising non-circular shift consistency and lowering PosENet correlation on noise.

The strongest existing method for leakage removal is full convolution (Kayhan & van Gemert), applied to NAT's downsamplers and paired with each attention border rule. It is the one intervention with evidence that it removes coordinate exploitation outright, with a reported 26.4% accuracy and 31.1% consistency gain at 50 images per class on ImageNet. Its known costs are roughly 15% lower throughput and, by the analogy to Islam et al., likely damage to position-dependent tasks.

For the shift-consistency axis, a BlurPool-augmented NAT (Zhang) is the strongest local-jitter fix. The reported gain for BlurPool on ResNet-50 is 1.3–2.1 consistency points and +0.7–0.9 top-1. On the position-prior axis, we reproduce the zero-padding-ablated saliency setting on PASCAL-S and DUT-S, where the CNN reference falls from 0.66 to 0.32 F-measure.

Two ablations isolate the ingredients. The first is masked truncation without entropy correction, which tests whether the correction matters. The second is the full redesign with the coordinate gate removed during training, which tests whether the explicit channel is what rescues position tasks.

All variants share data, augmentation, epochs, and a fixed hyperparameter sweep of three learning rates. The factorial study runs at a small NAT scale for 100 ImageNet epochs. The top three configurations are rerun at NAT-Tiny with the full published recipe and then fine-tuned on ADE20K, COCO, and PASCAL-S under identical detector and segmenter heads.

Why not just…?

Why not just turn off relative position bias in NAT? This is the most serious threat. If lopsided offsets are the leakage channel, removing the bias might fix most of it for free. It is one cell of the factorial, so the study settles the question directly. If turning off the bias alone drives PosENet correlation to the same floor as our redesign, at equal accuracy, we will report that as the main finding and drop the entropy-corrected truncation. The coordinate-gate question survives either way, because removing leakage still costs position-dependent tasks their prior.

Why not just reflect-pad the token map before sliding-window attention? Reflection is the cheapest neutral-looking option, but the evidence says it is not neutral. Islam et al. 2021 and Zhu et al. find that reflection-padded CNNs still encode coordinates. Garcia-Gasulla et al. find that switching to reflection shifts prediction odds by more than 7% for over 100 ImageNet classes. Reflection is included as an attention border rule, so its leakage in attention specifically is measured rather than assumed.

Why not just train with heavy translation and crop augmentation? Azulay & Weiss find that augmentation-trained CNNs still break under one-pixel shifts. Augmentation also does not remove the border landmark, because the frame edge is present in every crop. We apply the same strong random-resized-crop recipe to all variants, so any gap we find exists on top of augmentation.

Why not just append coordinates to stock NAT, CoordConv-style? This gives position tasks their prior, but it leaves the accidental channel open. The model then has two redundant position sources, and zeroing the explicit one at test time would not remove its reliance on position. We run this variant and apply the test-time gate-off check. The prediction is that stock NAT plus coordinates loses little when the gate is zeroed, because it still has the border channel, while our redesign loses its position-task performance when the gate is zeroed, because the gate is its only position source.

Proposed Experiments

Shift consistency is measured under natural boundaries, never circular ones. Following the protocol of Zhang, we take two crops offset by $\delta \in \{1, 2, 4, 8, 16, 32\}$ pixels from a larger image and record whether the predicted class agrees. Integer and sub-pixel offsets are reported separately.

Leakage is measured by PosENet Spearman correlation on Gaussian noise and on natural images, and by the KS border-aware fraction. Spatial robustness is measured with SI-Score location heatmaps, summarized as the accuracy drop from the center row to the edge rows. Task accuracy is measured by ImageNet top-1, ADE20K and Cityscapes mIoU, COCO box AP, and PASCAL-S and DUT-S F-measure. Bosch Small Traffic Lights [email protected] serves as a blind-spot check. On that benchmark, Alsallakh et al. report that mirror padding lifts a CNN from 49.58% to 57%.

We commit in advance to three predictions:

  • Stock NAT will show substantial PosENet correlation on noise. DiNAT will leak more, because dilation places a larger fraction of tokens in border-affected windows.
  • Removing the coordinate gate will cost the redesign most of its saliency F-measure, while zeroing coordinates appended to stock NAT will cost little.
  • The redesign will beat stock NAT on shift consistency at $\delta \geq 8$, the global regime where Lee et al. show anti-aliasing does not help.

The decisive outcome is a single model that matches stock NAT on ImageNet top-1 and PASCAL-S. It must also have lower noise-input PosENet correlation, show higher large-offset shift consistency, and lose its saliency advantage only when the gate is zeroed. Each kind of failure is informative. If masked truncation underperforms re-centering on accuracy, re-centering's $k^2$ keys matter more than its leakage costs. If no attention border rule moves PosENet correlation while stem padding does, the leakage lives in the convolutions and attention is a bystander.