Revisit-Invariant Linear Probes for Few-Label, Out-of-Region Satellite Mapping

When only a few hundred labeled satellite tiles exist, and all of them come from a handful of regions, the usual approach is to freeze a pretrained Earth-observation model and fit a linear classifier on its features. This is called a linear probe: a single linear layer trained on top of a frozen encoder. Probes are cheap and hard to overfit badly, yet they still fail when deployed in a new region. We argue that much of this failure is a shortcut, meaning the model relies on an easy signal that happens to correlate with the label rather than on the thing the label describes. With few labels from few places, the probe is free to put weight on feature directions that encode how an image was acquired rather than what is on the ground. Examples are sun angle, haze, season, and sensor or resampling quirks. Those directions hardly vary in the small training set, then swing widely in a new region. Satellites supply a free, label-free way to find these directions: the same patch of ground is photographed again every few days. Any variation in a frozen embedding across revisits of an unchanged location is, by construction, variation that should not change the label.

The proposal is to estimate this "revisit subspace" from unlabeled archive imagery and to train linear probes that are penalized for being sensitive to it. The penalty strength is chosen without target labels, by checking whether predictions stay stable across revisits in the unlabeled target region. In short: the orbit gives us real augmentations for free, and we apply them in feature space at probe time rather than during pretraining. This works on any frozen geospatial foundation model and requires no retraining. It also yields a diagnostic, namely how much of a probe's weight lies in the revisit subspace, which we hypothesize predicts the size of its out-of-region drop.

Core Claim

Our claim is that, in the label-scarce regime, the out-of-region gap of linear probes on frozen Earth-observation encoders is concentrated in a low-dimensional set of embedding directions. These directions vary across revisits of the same unchanged location. Removing the probe's sensitivity to them should recover a substantial part of that gap without target labels. Doing so does not discard the regional differences in land cover that the label actually depends on. This is what separates the approach from simply re-centering features per region.

Let $f$ be the frozen encoder. Let $\Sigma_w$ be the covariance of embedding differences $f(x_{\ell,t}) - f(x_{\ell,t'})$ over pairs of images of the same location $\ell$ at dates $t, t'$ with no land-cover change. The probe with weights $W$ minimizes

$$ \min_{W,b}\; \frac{1}{n}\sum_{i=1}^{n} \mathcal{L}\big(W f(x_i)+b,\; y_i\big) \;+\; \lambda\, \mathrm{tr}\big(W \Sigma_w W^\top\big), $$

Here $n$ is the number of labeled examples, $\mathcal{L}$ is cross-entropy, and $\lambda \ge 0$ trades training fit against revisit sensitivity. The penalty is the expected squared change in the probe's logits between two acquisitions of the same place. It is a Mahalanobis-style ridge penalty: it leaves directions that vary only between locations untouched. A hard variant projects out the top-$k$ eigenvectors of $\Sigma_w$ relative to the between-location covariance, which amounts to linear discriminant analysis with "location" as the class. We then fit the probe on the remaining subspace.

Relation to Prior Work

Four lines of work motivate the gap. First, Shortcut learning in deep neural networks argues that shortcuts stay invisible under i.i.d. validation. Earlier self-supervised work documents concrete low-level shortcuts, such as lens chromatic aberration in Carl Doersch et al. and resampling fingerprints in Mehdi Noroozi et al.. Second, Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning reports out-of-region drops of up to 30% and warns that color-jitter-style fixes destroy calibrated reflectance. None of these papers asks which directions in a frozen satellite embedding a few-label probe actually uses.

Third, Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution shows that probing preserves pretrained features better than full fine-tuning under shift, and that probing first then fine-tuning (LP-FT) helps. That argument assumes the preserved features are good. If they carry acquisition shortcuts, a probe preserves the shortcut. Our method acts on exactly this weakness. Fourth, Fine-tune Smarter, Not Harder benchmarks probing, LoRA, and full fine-tuning on geographic hold-out sets. It finds LoRA best and metadata inputs unhelpful, but does not look at which feature directions drive the gap.

Two papers supply the method, and we depart from each in a specific way. Don't forget the nullspace! shows that out-of-distribution errors arise when test features move along directions the training data never exercised, and that projecting onto a learned basis helps. It finds that basis from labeled training data and picks its rank by a training-accuracy heuristic. Under 1% labels from few regions, the labeled data's unexercised subspace is enormous and uninformative. We instead identify the harmful directions from unlabeled revisit structure, and we choose strength by target-side revisit consistency.

Seasonal Contrast (SeCo) uses revisits as positive pairs to make pretraining invariant to season and illumination. We borrow its insight but apply it post hoc, at probe time, to encoders such as masked autoencoders (SatMAE, Prithvi-EO-2.0) that were never trained to be revisit-invariant. Change-Aware Contrast supplies the idea we need for filtering revisit pairs that contain real land change.

This gives a falsifiable prediction. Gains should be largest for masked-autoencoder and ImageNet encoders, and near zero for SeCo, whose features should already have small $\Sigma_w$ along label-predictive directions. If SeCo gains as much as the others, the revisit-invariance explanation is wrong.

Research Program

The first component is a diagnostic study. For each frozen encoder (SatMAE, Prithvi-EO-2.0, Clay v1, SeCo, and an ImageNet ResNet-50) we build $\Sigma_w$ from unlabeled revisit pairs. The pairs are drawn from archive imagery at the benchmark locations, outside the labeled acquisition dates, and filtered to unchanged locations using a change score in the spirit of CACo. We then fit ordinary probes at 1%, 5%, 10%, and 100% of training-region labels. For each probe we measure the fraction of weight energy lying in the top revisit eigen-directions, and we correlate it with the in-region minus out-of-region accuracy gap across encoders, label budgets, and random seeds. To tie these directions to specific shortcuts, we regress their coordinates on acquisition metadata (day of year, sun zenith, view angle, cloud and haze fraction, processing baseline) and on high-frequency image energy. This tells us whether they are physically interpretable nuisance, the satellite analogue of the documented chromatic and resampling shortcuts.

The second component is the method. It consists of the penalized probe above, its hard-projection variant, and a label-free rule for choosing $\lambda$ or $k$. The rule takes unlabeled revisits of target-region locations and selects the smallest penalty at which the probe's prediction flip rate across dates falls to a plateau. For segmentation tasks the probe is a linear head on patch tokens, and the penalty uses token-level revisit differences.

We handle one known confound explicitly. For flood mapping, the label itself changes between revisits. $\Sigma_w$ must therefore be built only from dry-period pairs, and we test the sensitivity of the results to this choice.

Baselines

The naive reference is a standard linear probe on the frozen encoder, with L2 regularization tuned by cross-validation within the training regions. This is what practitioners do today, and it is what Seasonal Contrast and Tile2Vec report for label efficiency. Beating it by a consistent margin on held-out regions, at 1% and 5% labels, is the minimum bar.

The strongest existing competitor for geographic transfer is LoRA adaptation of the foundation model, as in Fine-tune Smarter, Not Harder. That paper reports 87.57% mIoU on the Sen1Floods11 geographic hold-out set, against 82.07% for full fine-tuning. We run LoRA through TerraTorch, the library that paper used, on identical label subsets and seeds, with the same hyperparameter search budget as our method. Model selection for LoRA uses only training-region validation, because target labels are unavailable in deployment.

We also run LP-FT from Fine-Tuning can Distort Pretrained Features, which reports about 10% better out-of-distribution accuracy than fine-tuning across its benchmarks. We add the Optimized Basis projection from Don't forget the nullspace!, which reports raising TerraIncognita accuracy from 48.7% to 51.9%; here it is fit on our labeled subsets with its accuracy-threshold rank rule. We found no reported number on Lacuna for LoRA, LP-FT, or the nullspace projection at 1% labels under geographic holdouts. Our runs will establish those numbers.

Two claims are needed for the proposal to count as a success. First, the penalized probe must beat the plain probe and the nullspace projection on out-of-region metrics at every scarce label budget. Second, at 1% labels it must match or exceed LoRA while training only a linear head.

The key ablations isolate the revisit subspace itself. We replace $\Sigma_w$ with, in turn, a random subspace of equal dimension, the top principal components of pooled unlabeled features, and the between-region covariance. We also compare our label-free choice of $\lambda$ against an oracle choice that uses target labels, which bounds how much the selection rule costs.

Why not just…?

Why not just use heavy augmentation such as color jitter or RandAugment when fitting the probe? Synthetic photometric distortions break the physical relationships between spectral bands, as the Mission Critical position paper argues. Debiased Learning for Remote Sensing Data reports that replacing generic augmentations with geometry-only ones improved accuracy by 5.2%. Revisits are real photometric augmentations with the physics intact, so our method is the principled version of this idea. We still include a jittered-probe baseline to confirm the point on our splits.

Why not just re-center features per region, or use test-time normalization? This is a genuine threat, because it needs no labels and removes mean regional shift. It also removes real differences in class frequency and land-cover makeup between regions, and it ignores nuisance variation within a region, such as date and haze. We test it directly by building target regions whose class priors differ from training. If per-region centering matches our method there, the revisit subspace adds nothing, and we would report that.

Why not just average embeddings over multiple dates at test time? For static land-cover tasks with dense time series, this may work about as well as our method, and we include it as a baseline on fMoW-WILDS. It does not stop the probe from learning on nuisance directions during training. It is also unavailable for single-date event mapping such as floods, which is where we expect our method to matter most.

Why not just use unsupervised domain adaptation methods such as domain-adversarial training (DANN) or correlation alignment (CORAL)? WILDS and WILDS 2.0 found that such methods rarely beat plain training on real shifts, and that they need target-side tuning. We include CORAL on frozen features as a cheap check. Our label-free consistency rule is designed to address the tuning problem those papers identify.

Proposed Experiments

We use two benchmarks. fMoW-WILDS provides region-held-out land-use classification with multi-date imagery per location, from which revisit pairs come directly. We report average and worst-region accuracy; no specific worst-region number is reported on Lacuna. Sen1Floods11, under the geographic hold-out protocol of Fine-tune Smarter, Not Harder, provides flood segmentation measured by mIoU, with dry-period revisits pulled from the public Sentinel archive. We add a label-scarce GEO-Bench classification task (GEO-Bench) to check that the penalty does not hurt in-distribution accuracy. Label budgets are 1%, 5%, 10%, and 100% of training-region labels, with five seeds and bootstrap confidence intervals.

Three checks decide the question. First, the fraction of probe weight in the revisit subspace should correlate with the out-of-region gap across encoders and budgets. Second, the random-subspace and pooled-principal-component ablations should recover much less of the gap than the true revisit subspace. Third, the gains should be ordered by encoder: masked-autoencoder and ImageNet encoders should benefit substantially and SeCo little. If all three hold, and the method matches LoRA at 1% labels on Sen1Floods11, the case is made that few-label geographic failure is largely an acquisition shortcut, and that the orbit itself supplies the cure. If per-region centering or temporal averaging matches the method, we will report that the effect is a regional mean shift rather than a revisit-specific shortcut.