Teaching Adapted Backbones to Ignore the Tile Grid

Very large images, such as 5000×5000 aerial photos or whole-slide pathology scans, do not fit through a modern vision backbone in one pass. Practitioners cut them into tiles, run each tile separately, and stitch the predictions back together. The stitched map then shows seams: buildings split in two at tile edges, class flips along straight lines, and lower confidence near borders. The usual fix is to run overlapping tiles and blend them, which multiplies inference cost by two to four times. This proposal treats the seam as a broken symmetry. The correct segmentation of a scene should not depend on where we happen to place the tile grid. We therefore train a lightweight adapter on a frozen pretrained backbone with one extra requirement. Run the same region twice under two grid offsets, once with a pixel near a tile edge and once with that pixel deep inside a tile, and make the network's internal features at the edge agree with its features in the interior.

In two sentences: seams arise because the network sees less context near tile edges and was never asked to behave the same there as in the interior. We add a label-free, layer-wise consistency loss across tile-grid offsets during adaptation, so that a single non-overlapping pass at test time approaches the quality of overlap-and-blend inference. Because the loss needs no labels, it can also adapt a backbone to unlabeled imagery from a new city or sensor.

Core Claim

Tile-and-stitch inference is a symmetry problem: the stitched prediction should be equivariant to translations of the tile grid. Prior symmetry work has either guaranteed this exactly under restrictive architectures or studied different symmetries. Soft, internal-feature regularization of grid-offset equivariance, applied only to the small set of parameters trained during adaptation, should remove most of the seam error of a frozen foundation backbone at single-pass cost. It should do so better than output-level consistency or augmentation alone.

The closest work is How Shift Equivariance Impacts Metric Learning for Instance Segmentation, and it motivates the proposal. That paper proves that U-Nets with pooling are only periodically shift-equivariant. It then derives "golden rules" for tile sizes and output crops under which stitched inference matches whole-image inference. Those rules depend on valid (unpadded) convolutions, which in practice means each input tile carries a context margin beyond the output it produces, so some redundant computation remains. They also do not carry over to Vision Transformers.

Reviving Shift Equivariance in Vision Transformers also motivates the proposal. It shows that patch embedding and positional encodings break shift equivariance in transformers. It restores equivariance architecturally with polyphase anchoring and circular padding, but only for classification. The distinction matters for us: even a perfectly shift-equivariant network produces seams, because a pixel at a tile edge sees truncated context. The equivariance that matters for tiling is equivariance of the whole tile-then-stitch procedure, not of the network on a single tile. No paper in the file regularizes that.

The method comes from EQ-Reg (A Regularization-Guided Equivariant Approach for Image Restoration). EQ-Reg enforces rotation equivariance through a loss on intermediate CNN feature maps rather than through rigid group convolutions. On DeepLesion it reports 43.04 dB PSNR, against 42.38 dB for the unregularized DICDNet and roughly 41–42 dB for strictly equivariant networks. We reuse its central lesson, that a soft layer-wise constraint beats both hard architecture and output-only supervision. We transfer it from rotations in restoration to grid offsets in adapted segmentation.

Research Program

Let $x$ be a large image and $\delta$ an offset of the tile grid. Let $F^{\ell}_{\delta}(x)$ be the stitched feature map at adapter layer $\ell$, where each tile is processed independently and the results are placed back in image coordinates. During training we sample a crop large enough to hold several tiles. We pick two offsets: $\delta_e$ puts a band of pixels within $k$ pixels of a tile edge, and $\delta_i$ puts the same pixels at least $k$ pixels from any edge. The regularizer is

$$ \mathcal{L}_{\text{grid}} = \sum_{\ell} \lambda_\ell \sum_{p} m_k(p)\,\big\| F^{\ell}_{\delta_e}(x)[p] - \mathrm{sg}\big(F^{\ell}_{\delta_i}(x)[p]\big) \big\|^2 , $$

where $m_k(p)$ is 1 for pixels in the edge band under $\delta_e$ and 0 elsewhere, $\mathrm{sg}$ stops gradients so the interior view acts as teacher, and $\lambda_\ell$ weights each layer. The interior view has more context and is treated as the better estimate. The loss pulls edge behaviour toward interior behaviour rather than averaging the two, which is what overlap blending effectively does.

On labeled data the total loss adds standard segmentation cross-entropy. On unlabeled target imagery $\mathcal{L}_{\text{grid}}$ is used alone or alongside pseudo-labels. Pseudo-labels near contours are noisy, a known problem in source-free adaptation, and the interior teacher provides a label-free signal at exactly those locations.

The backbones are kept frozen: a Swin-B, as used by Uni-UVPT, and a plain ViT. Only a small adapter is trained, so the comparison isolates what the regularizer does to adaptation. We use two adapter types: the convolutional prompt generator with cross-attention interactors from Uni-UVPT, and a low-rank adapter. The work has three parts, described below.

The first part is a measurement study. Using frozen backbones, it quantifies seam error as a function of distance to the tile edge, tile size, and padding mode. It also checks how much of the error comes from missing context and how much from positional-embedding phase. The second part trains the regularizer and ablates it: which layers receive it, the band width $k$, and the stop-gradient direction. The third part extends the symmetry group from grid offsets to grid offsets combined with the eight flips and 90° rotations, which suits overhead imagery with no canonical orientation. This tests the file's observation that truncation geometry itself changes under rotation (Gap 1). The extension is secondary and will be reported as such.

Baselines

The naive reference point is the same frozen backbone and adapter, trained with standard random crops and run with non-overlapping tiles and no consistency loss. Every other method is measured relative to this seam error. Fairness here means identical tile size, backbone, adapter size, training iterations and labeled data.

The strongest inference-time competitor is overlapping sliding windows with Gaussian blending, at 25% and 50% overlap, applied to the naive model. It is the standard practical fix and needs no retraining. We report it at its true cost of extra forward passes, and also at matched compute by giving our model the same pass budget. To beat it, our single-pass model must match its mIoU and seam-band mIoU within a small margin while using one pass instead of two to four. The model must also clearly exceed it once both use the same number of passes.

The strongest training-time competitor is distillation from a sliding-window teacher. The naive model's overlap-blended outputs serve as soft targets for the adapter, a sensible output-level answer to seams. We run it with the same unlabeled images and the same number of updates.

The key-ingredient ablation is an augmentation-matched control. It samples exactly the same pairs of grid-offset views and applies supervised loss to both, but drops the feature-consistency term. This separates "seeing more truncation configurations" from "being asked to be consistent across them." A second ablation applies $\mathcal{L}_{\text{grid}}$ to output logits only rather than to intermediate features, which tests the EQ-Reg claim in this setting.

As reference points for absolute accuracy we cite architectures designed to avoid tiling. BPT groups similar patches into content-aware tokens and adds Canny-edge auxiliary supervision. It reports 77.1% mIoU on Inria Aerial and 76.6% on DeepGlobe. ISDNet pairs a deep branch on a downsampled image with a shallow full-resolution branch. It reports 74.23 mIoU at 6.90 FPS on Inria and 73.30 mIoU at 27.70 FPS on DeepGlobe. These are fully trained specialized models, not frozen-backbone adapters, so we do not claim to beat them outright. Our result is meaningful if a frozen foundation backbone with our adapter lands in their range at comparable throughput. We will also re-run ISDNet under our data splits where possible.

Why not just…?

Why not just use overlapping tiles and blend? This is the real threat on accuracy, and the proposal is built to measure it head-on. The file documents its cost: ISDNet reports patch-based crop-and-merge pipelines running up to 172× slower than its single-pass design. Rumberger et al. report that heuristic overlap does not remove false splits unless tiling respects the network's periodicity. Neither result shows that blending leaves accuracy on the table for a ViT, though. If our single-pass model only matches blending at 25% overlap, the contribution is a speed one, and we will say so. If blending applied to our model gives no further gain, that is evidence the network has internalized the context deficit.

Why not just augment with random crop positions? Random crops already expose the model to objects truncated at every position. EQ-Reg's evidence is that augmentation supervises only the output and leaves internal features unconstrained: DICDNet trained with augmentation reaches 42.38 dB against 43.04 dB with the internal loss. That evidence comes from restoration and from rotations, so it does not settle the question here. The augmentation-matched control above does, because it sees the identical pairs without the consistency term.

Why not just make the transformer shift-equivariant architecturally? Polyphase anchoring from Reviving Shift Equivariance in Vision Transformers fixes sensitivity to sub-patch phase, raising ImageNet consistency to 100%. It does nothing about edge pixels seeing less context than interior pixels, which we expect to be the dominant source of seams. The measurement study tests this by splitting seam error into a phase-only component and a truncation component. We measure the phase-only component by shifting the grid by sub-patch amounts within one large tile, and the truncation component by the edge-distance profile. If phase dominates, polyphase anchoring is the better fix and we would adopt it.

Why not just avoid tiling with a dual-branch network like ISDNet or Hyden? Dual-branch designs are an alternative, but they require training a dedicated architecture rather than adapting a frozen foundation backbone. Hyden is one such design, pairing a frozen ViT on a downsampled image with a full-resolution CNN. The file also notes these designs may have their own cross-branch misalignment under transformations (Gap 4). We treat them as accuracy references, not as replacements for the adaptation setting we study.

Proposed Experiments

The primary benchmarks are Inria Aerial and DeepGlobe land cover. Inria's held-out-city protocol doubles as the adaptation test: labels come from training cities, and the regularizer runs on unlabeled imagery from the held-out cities. We report six metrics:

  1. Overall mIoU.
  2. Seam-band mIoU, computed within $k$ pixels of tile edges, and the gap between it and interior mIoU.
  3. Grid-offset disagreement: the fraction of pixels whose label changes when the tile grid is shifted.
  4. Building false-split rate: connected footprints broken at seams, measured on Inria.
  5. Forward passes per pixel.
  6. Throughput.

A secondary check uses GTA5→Cityscapes source-free adaptation with Swin-B at deliberately small tile sizes, so that seams matter. Uni-UVPT's reported 56.9% mIoU there is the reference for the adapter without our loss, and our first task is to reproduce it at full resolution.

Four results are decisive. First, the regularized single-pass model must close most of the seam-band gap of the naive model and reach within about one mIoU point of 50%-overlap blending. Second, it must beat the augmentation-matched control and output-level distillation on seam-band mIoU, or else the layer-wise consistency term is not doing the work. Third, on held-out Inria cities, the label-free version must improve seam metrics without lowering interior mIoU. Fourth, the dihedral extension must reduce disagreement under rotated grids without lowering mIoU on oriented structures such as roads. Failure on the third or fourth check would narrow the claim to supervised seam removal, but would not sink it.