Phase-Split Tokenization: Overlapping Patches for Frozen Vision Transformers on Aerial Imagery

Vision Transformers (ViTs), the image models that cut a picture into a fixed grid of 16×16-pixel squares and treat each square as a "word," have a blind spot that matters a great deal in satellite imagery. More than half of the objects in aerial benchmarks are 10 to 50 pixels across (DOTA). A 15-pixel car that happens to sit on a grid line gets split across two to four squares, and no single token ever sees the whole car. How well the model detects that car then depends on where the arbitrary grid happens to fall. We propose a fix that adds no new tokenizer and leaves the pretrained backbone frozen. We run the frozen ViT four times on copies of the image shifted by half a patch (0 or 8 pixels horizontally and vertically), then interleave the four token maps into a single feature map with an 8-pixel stride. That map is exactly what an overlapping patch embedding with kernel 16 and stride 8 would produce, except that every token is computed with the original pretrained projection, at the image scale the backbone was trained on.

The core claim is that a large share of the remote sensing failure of frozen natural-image ViTs on dense tasks comes from grid-phase fragmentation of tiny objects rather than from the overhead viewpoint itself. We also claim that removing this fragmentation with phase-split tokenization lets a frozen backbone plus a small trained fusion module and head close much of the gap to full fine-tuning on both detection and segmentation. The proposal first measures the effect directly, then tests the fix against the cheap alternatives a reviewer would reach for first: upsampling the input, averaging predictions over shifted copies, and learning a new overlapping convolutional stem.

Why this is the right place to push

The overlapping-patch literature shows that letting neighboring tokens share pixels preserves boundaries. It does this in models trained end to end, not frozen ones. SegFormer uses overlapped patch merging (kernel 7, stride 4, then kernel 3, stride 2) in a hierarchical encoder trained from scratch. PVT v2 and CvT replace the linear patch projection with strided convolutions. CrowdFormer is the closest in spirit: it applies sliding-window patching to top-down crowd counting and reports that replacing its overlap blocks with plain 3×3 convolutions raised error on UCF-QNRF from 78.8 to 90.2 MAE. These papers supply the motivation, but none can be applied to a pretrained foundation model without retraining. The research file names exactly this obstacle: swapping in an overlapping stem breaks the alignment with the pretrained projection and position embeddings.

The adaptation literature constrains what will work. Barbier et al. found that GLIP scores 0.04 mAP zero-shot on DIOR, that LoRA and other parameter-efficient methods do poorly with 2 to 5 labeled examples per class, and that full fine-tuning was needed (0.75 mAP with full data). They attribute the failure to entanglement between the vision and text branches. Uni-UVPT points to a different reading. Prompt tuning for segmentation only worked once the authors added a convolutional side branch with spatial structure, and it then reached 56.9% mIoU on GTA5→Cityscapes while training 12M–28M parameters. Our working hypothesis is that channel-only adapters such as LoRA cannot repair a spatial failure. Phase-split tokenization is the most direct test of that hypothesis, because it changes only where tokens sit and nothing about what the frozen weights compute.

The remote sensing literature supplies the stakes. RVSA pre-trains a plain ViT with 16×16 patches on MillionAID and adds rotated, variable-size attention windows, reaching 81.24% mAP on DOTA-V1.0. The authors concede that the plain model lags hierarchical models on segmentation (Potsdam, iSAID, LoveDA) and attribute this to lost fine spatial detail. Because RVSA keeps the non-overlapping 16×16 stem, its backbone is also a natural test bed. If phase-split tokenization helps an in-domain pretrained backbone as well as a natural-image one, the gain is architectural and not a domain-gap artifact.

Core Claim

Let $B$ be a frozen plain ViT with patch size 16. For offsets $(u,v) \in \{0, 8\}^2$, let $F_{u,v} = B(\mathrm{shift}_{u,v}(I))$ be the token grid computed on image $I$ shifted by $u$ pixels horizontally and $v$ pixels vertically. We build a stride-8 feature map $G$ by interleaving the four grids:

$$G(2i + v/8,\; 2j + u/8) = F_{u,v}(i, j).$$

Here $(i,j)$ index tokens in each 16-stride grid. Every token of $G$ comes from a full 16×16 window, and adjacent tokens overlap by half, so $G$ matches an overlapping patch embedding with kernel 16 and stride 8. The cost is four ordinary forward passes. With global attention over $L$ tokens per pass, the attention cost is $4L^2$. Upsampling the image by 2× would put $4L$ tokens in a single pass and cost $16L^2$, because attention cost grows with the square of sequence length.

Interleaving alone is not enough. The four passes do not interact, so we train one lightweight cross-phase fusion layer: a few local attention blocks over 3×3 neighborhoods of $G$, initialized near identity. This layer lets a token see its half-shifted neighbors and resolve objects that were split in one phase but whole in another. A standard detection or segmentation head sits on top. The backbone stays frozen. The same frozen backbone with the same fusion design serves oriented detection and semantic segmentation, which is the cross-task part of the claim. The two tasks should agree: the one representation fix should help both.

Research Program

The first stage is a diagnostic. It is valuable even if the fix fails, and it decides whether the fix is worth building. On DOTA-V1.0 and DIOR, we train a standard head on a frozen backbone, then evaluate on every one of the 64 integer-pixel shifts of each test crop within a 16×16 cell. For each small object (10–50 px), we record detection confidence and IoU as a function of where its center falls relative to the patch grid. We call the resulting spread across shifts the grid-phase sensitivity and summarize it as the variance of small-object AP over shifts, split by object size. We run this for a natural-image plain ViT-B, a natural-image ViT-B whose head is fine-tuned with LoRA, and the RVSA in-domain backbone. If phase sensitivity is small, about a point of small-object AP or less, the fragmentation hypothesis is wrong for this setting and we stop. If it is large and concentrated in the smallest size bin, the hypothesis survives and predicts which objects the fix should help.

The second stage builds phase-split tokenization with cross-phase fusion and runs it for oriented detection on DOTA-V1.0 and DIOR-R, horizontal detection on DIOR, and segmentation on iSAID, Potsdam and LoveDA. Each runs at full data and at the 10-shot and 20-shot-novel splits that Barbier et al. used on DIOR. Position embeddings need no change, because each phase pass sees an ordinary image of the original size. The pixel shift is handled by padding the image with reflected edge pixels.

The third stage asks whether two phases are enough and where overlap stops paying. We vary the number of phases (1, 2 along the diagonal, 4, and 16 at stride 4) and measure the cost against small-object AP. We also test whether fusion that is aware of object orientation helps, by combining phase-split tokens with RVSA's rotated windows on the RVSA backbone.

Baselines

The naive reference point is a frozen backbone with a single phase and the same trained head and neck. This is our method with the number of phases set to one. Every other number is reported relative to it, and the diagnostic in the first stage is computed on it.

The strongest existing reference on the natural-to-aerial question is full fine-tuning of the backbone, following Barbier et al.. That paper reports 0.75 mAP on full DIOR and 0.37 mAP at 10-shot for fully fine-tuned GLIP, and 0.55 mAP for 20-shot novel classes. Those numbers come from a Swin-T backbone, so we will not compare against them directly. Instead we rerun full fine-tuning on our plain ViT with the same head, data, schedule and hyperparameter search budget as our method. Beating it means matching its full-data mAP within one point while training under 10% of the parameters, or exceeding it at 10 shots. For in-domain pretraining, the reference is RVSA, which reports 81.24% mAP on DOTA-V1.0 (RVSA). That figure is a fully fine-tuned result. We will add phase-split tokenization on top of the frozen RVSA backbone and also of a fine-tuned one, and count a gain on the small-vehicle, ship and storage-tank classes as the win. No reported result was found on Lacuna for RVSA's DIOR-R mAP or for any method's numbers on iSAID, Potsdam or LoveDA, so those comparisons use our own reruns.

The parameter-efficient competitors are LoRA on the frozen backbone and a Uni-UVPT-style convolutional side branch with cross-attention interactors (Uni-UVPT), trained under supervision rather than source-free. Both get the same head and a trainable parameter budget equal to or larger than ours.

The ablation that isolates the key ingredient keeps the four passes but drops the overlap. It feeds four copies with the same zero shift, so compute is identical, or it uses four random shifts by whole multiples of 16 pixels. If that matches our method, the gain came from compute or ensembling, not from overlap. A second ablation removes the cross-phase fusion layer and keeps plain interleaving. This tells us whether the phases need to talk to each other.

Why not just…?

Why not just upsample the image 2×? This is the most serious threat. Upsampling also makes tiny objects span more tokens, and SegFormer-style models already work at stride 4. It differs from our method in two ways. It costs 16 times the global attention compute instead of 4, and it shows the frozen projection textures at twice their natural size, a scale mismatch that may hurt a frozen model in a way it would not hurt a fine-tuned one. We will run it at matched windowed-attention compute and matched trained parameters. If 2× upsampling equals phase-split tokenization on small-object AP, the honest conclusion is that resolution, not overlap, is the lever, and the remaining contribution is the 4× attention saving.

Why not just average predictions over shifted copies at test time? Test-time shift averaging needs no training and also smooths over the grid phase. It cannot, however, produce a stride-8 feature map, and it cannot recover an object that every phase sees only in part. That second property is precisely what the cross-phase fusion layer targets. The diagnostic in the first stage tells us how often small objects are split in all phases versus only some. Running shift averaging as a baseline at the same four-pass cost settles whether learned fusion adds anything beyond averaging.

Why not just learn an overlapping convolutional stem, as SegFormer and Uni-UVPT do? A learned stem with stride smaller than its kernel is the obvious combination of these lines of work. It must either replace the pretrained projection, which breaks its alignment with the frozen layers above, or run as a side branch, which then has to be trained from labels that may be scarce. Barbier et al.'s few-shot results suggest that low-data regimes are exactly where newly trained spatial modules struggle. Phase-split tokenization trains nothing at the tokenizer. The comparison at 10 shots, and with full data, will show whether keeping the pretrained projection matters more when labels are scarce.

Why not just use LoRA or full fine-tuning? Barbier et al. already report that LoRA fails at 2–5 shots. Full fine-tuning works but gives up modularity: one task-specific copy of the backbone per dataset, and no shared backbone across detection and segmentation. If phase-split tokenization plus LoRA beats LoRA alone, that supports the reading that LoRA's failure was spatial. If full fine-tuning on a single phase still clearly wins at full data, we report that our method's value is confined to the low-data and multi-task regimes.

Decisive Checks

The idea stands or falls on three results. First, the diagnostic must show substantial grid-phase sensitivity for small objects on a frozen natural-image ViT. Second, phase-split tokenization must beat both 2× upsampling and shift averaging at matched compute on small-object AP on DOTA-V1.0 and DIOR-R. Third, the same frozen backbone and fusion design must improve iSAID segmentation, where RVSA's authors located the weakness of plain ViTs. If the gain shows up only in detection and not in segmentation, or vanishes against upsampling, we will say so and report the diagnostic as the main finding.