Segmentation models for satellite and aerial imagery label each pixel as road, building, forest, water, and so on. They often learn the typical arrangement of a scene rather than what each surface actually looks like. A model trained on cities learns that buildings line roads in grid patterns, and when it is moved to farmland, where layouts differ, it mislabels pixels that look perfectly ordinary up close. We propose a simple fix. Cut the training image into tiles and shuffle them, apply the same shuffle to the label map, and train on the pair. Classification cannot do this, because shuffling destroys the label. For segmentation, shuffling is a permutation that can be undone exactly, so the label moves with the pixels. The model keeps its full input size and its ability to localize, but it can no longer rely on long-range layout. The tile size sets how much surrounding context the model is allowed to use.
This matters most when labels are scarce and the deployment region differs from the labeled one, which is the usual situation in Earth observation. The same undoable shuffle also gives a second use. On unlabeled target imagery, the model predicts on the original image and on a shuffled copy, and the shuffle is reversed on the second prediction. Where the two disagree, the prediction on the original image was probably driven by layout priors carried over from the source domain. Those pixels are the ones to leave out of self-training. We study this on top of masked-image-modeling backbones, meaning encoders pretrained by hiding most of an image and reconstructing it. Such encoders are reported to organize features around local structure, which makes them good candidates to benefit from context limits rather than be hurt by them.
Core Claim
Spatial disruption has so far been used almost entirely for whole-image tasks, and the research file names its "conflict with dense localization" as an open problem. We claim that for segmentation there is no such conflict if the disruption is treated as something the output should follow (shuffle the input, shuffle the answer) rather than something the output should ignore. Concretely, with $\pi$ a random permutation of $k \times k$ tiles, $x$ an image, $y$ its label map, and $S$ the set of pixels within $w$ pixels of a tile seam, the supervised loss is
$$\mathcal{L}_{\text{shuf}} = \sum_{i \notin S} \ell\big(f(\pi(x))_i,\ \pi(y)_i\big),$$
where $f$ is the segmentation network and $\ell$ is per-pixel cross-entropy. Excluding the seam band stops the model from being penalized at artificial edges it was never meant to learn. Our specific, testable prediction has two parts. First, mixing this loss with ordinary training at a tile size near the scale of individual land-cover objects will narrow the source-to-target mIoU gap on urban-to-rural and city-to-city transfer. Second, it will do so more for masked-image-modeling backbones than for contrastive ones. The second part is falsifiable in the opposite direction, because Contrastive Learning Relies More on Spatial Inductive Bias Than Supervised Learning finds contrastive models lose 27% ImageNet accuracy under patch shuffling versus 10% for supervised ones. Contrastive backbones may therefore gain more from being weaned off layout, or may simply break.
Relation to Prior Work
Four papers motivate the idea. Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning (ReCIT) shows that breaking token continuity pushes Vision Transformers toward small, transferable patterns and improves few-shot transfer to EuroSAT. It works only for classification, uses supervised natural-image pretraining, and pays a cost on the source domain. We move the idea to dense outputs and measure that source-domain cost directly. Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation finds ViTs keep about 80% ImageNet accuracy on fully shuffled patches and uses shuffled images as negatives the model should be uncertain on. We do the opposite and treat shuffled images as positives with shuffled labels, because in segmentation the local evidence is supposed to be enough. Generalized Domain Adaptation uses grid shuffling to erase class identity while keeping domain style. It also documents the grid-size sensitivity we must face: coarse grids leak semantics and fine grids destroy structure.
The closest methodological precedent is patch-swapping augmentation for defocus blur detection by Ning Zhang et al.. That work is already a dense task where swapped patches carry their labels, used to stop semantic cues from predicting blur. We therefore do not claim label-following shuffles as new in themselves. The new parts are three: applying them to land-cover segmentation under geographic domain shift with scarce labels, using shuffle agreement to filter pseudo-labels on unlabeled target data, and analyzing how the effect depends on the pretraining objective. Patch-Mix Transformer for Unsupervised Domain Adaptation mixes source and target patches for classification and explicitly leaves dense extension open. Our cross-domain tile variant, described below, is that extension in its simplest form.
On the application side, LoveDA supplies the urban/rural benchmark. It reports that self-training methods beat adversarial ones and that background clutter is a main error source. Tile shuffling targets exactly the context-driven part of that confusion. ST-DASegNet separates style from content in feature space with dual students, adversarial losses, and a slowly updated teacher model. We intervene at the input instead, with no extra networks. The MIMRS survey reports masked-autoencoder backbones leading contrastive ones on remote-sensing segmentation and notes that none use spatial disruption. Continual Learners are Incremental Model Generalizers argues that masked modeling organizes features around local structure, which supplies the mechanism for our backbone prediction. Seasonal Contrast provides the contrastive remote-sensing backbone we compare against.
Research Program
The first step is diagnostic. For each frozen-or-finetuned backbone, we evaluate segmentation on test images shuffled at tile sizes from 16 to 256 pixels, reverse the shuffle on the predictions, and score them. The drop from unshuffled performance is a "layout reliance" curve for each backbone. We compare three backbone families: masked-autoencoder ViTs, the contrastive SeCo ResNet-50, and ImageNet-supervised models. We then ask whether in-domain layout reliance predicts the size of the cross-domain drop. If it does not, the premise is weak and we stop before the method work.
The second step is the method on labeled source data. Each minibatch mixes unshuffled and shuffled views at a ratio $\rho$. The tile size is either fixed or drawn from a range, borrowing ReCIT's warm-up schedule that moves from fine to coarse tiles. Tile edges are aligned to the 16-pixel token grid so seams fall between tokens rather than inside them.
The third step targets the unlabeled target domain. We take a standard teacher-student self-training loop of the kind LoveDA found strongest. A target pixel's pseudo-label is kept only if the teacher's prediction on the original image agrees with its reverse-shuffled prediction on a shuffled copy. The idea is that labels which survive loss of context rest on local evidence.
A fourth variant builds some tiles of a shuffled image from target images. Only source tiles carry supervised loss, and target tiles take the filtered pseudo-labels. This is PatchMix without learned weighting.
Label scarcity enters through source label budgets of 1%, 5%, 10%, and 100% of LoveDA-Urban and Potsdam training tiles, each sampled with three fixed seeds.
Baselines
The naive reference point is ordinary supervised fine-tuning of each backbone on the labeled source split with standard flips, rotations, and crops. It is evaluated directly on the target domain, and target-only training gives the oracle ceiling. This shows how large the transfer gap is before any intervention.
The strongest existing competitor in the file is ST-DASegNet (Self-Training Guided Disentangled Adaptation). It was designed for exactly our Potsdam/Vaihingen and LoveDA settings and combines disentanglement, adversarial alignment, and teacher self-training. We rerun it from released or reimplemented code on our backbones where the architecture allows and on its original backbone otherwise, reporting both. The file gives no reported result for ST-DASegNet or for the self-training methods benchmarked in LoveDA (CBST, IAST), so no reported number is found on Lacuna for either, and we will rerun them rather than quote them. We also run plain teacher-student self-training with confidence-threshold pseudo-labels, CBST-style, as the direct parent of our third step.
To keep comparisons fair, every method uses the same backbone checkpoint, the same decoder, the same number of iterations and batch size, and the same budget of eight hyperparameter configurations tuned on a held-out source-only validation split. No target labels are used for tuning.
Beating the strongest competitor means a target mIoU gain over ST-DASegNet larger than the seed-to-seed standard deviation on LoveDA Urban→Rural at two or more label budgets. It also means no more than a one-point loss in source-domain mIoU, which is ReCIT's "no free lunch" concern made measurable.
The ablations that isolate the key ingredient compare the full method against four reduced versions: shuffling the image without the label (turning the shuffle into invariance, which should fail); removing the seam mask; replacing the agreement filter with a confidence threshold alone; and random tile mixing without shuffling.
Why not just…?
Why not just train on small random crops? Small crops also limit context, and this is the most serious threat. The differences are that shuffled images keep the token count and attention range the model sees at inference, and that shuffling breaks correlations between distant regions while still showing many local contexts per image. We will run crop training with crop size matched to tile size and identical compute. If crops match our target mIoU, the contribution shrinks to the agreement-filtered self-training and we will say so.
Why not just continue masked-autoencoder pretraining on the unlabeled target region? Domain-adaptive pretraining is cheap and plausibly strong, and the MIMRS survey's evidence that masked backbones transfer well makes it a real competitor. We include it as a baseline, and also stack our method on top of it. The claim survives only if the gains add up.
Why not just use stronger mixing augmentation or adversarial alignment? Feature-level alignment is what ST-DASegNet already does. The LoveDA benchmark found adversarial training weaker than self-training on background clutter, which suggests that aligning feature distributions does not remove layout priors. Our ablation separating tile mixing from shuffling tests whether breaking layout, rather than mixing appearance, is what helps.
Proposed Experiments
The main benchmarks are LoveDA Urban→Rural and Rural→Urban, and Potsdam↔Vaihingen, scored by target mIoU, per-class IoU, and boundary F-score. Boundary F-score checks that seams do not degrade edges. We also report source mIoU at each label budget.
Three checks are decisive. First, the diagnostic must show a positive correlation between layout reliance and cross-domain drop across backbones. Second, at a matched tile size, the method must beat matched-crop training on rural classes (agriculture, forest, barren) in LoveDA, with the gain largest at the 1% and 5% label budgets. Third, pseudo-labels kept by the agreement filter must be measurably more accurate against held-out target labels, used for analysis only, than confidence-thresholded ones at equal coverage.
We will additionally report whether the benefit depends on the backbone: larger for masked-autoencoder ViTs than for SeCo, or the reverse. Either outcome is informative about how pretraining shapes reliance on layout. Any negative outcome will be reported: a large source-domain loss, or no correlation between layout reliance and transfer drop, would show that for aerial segmentation context is signal rather than shortcut.
References
- Contrastive Learning Relies More on Spatial Inductive Bias Than Supervised Learning: An Empirical Study
- Revisiting Continuity of Image Tokens for Cross-domain Few-shot Learning
- Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentation
- Generalized Domain Adaptation
- Rethinking the Defocus Blur Detection Problem and a Real-Time Deep DBD Model
- Patch-Mix Transformer for Unsupervised Domain Adaptation: A Game Perspective
- LoveDA: A Remote Sensing Land-Cover Dataset for Domain Adaptive Semantic Segmentation
- Self-training guided disentangled adaptation for cross-domain remote sensing image semantic segmentation
- MIMRS: A Survey on Masked Image Modeling in Remote Sensing
- Continual Learners are Incremental Model Generalizers
- Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data