Self-supervised contrastive learning trains an image encoder by pulling two views of the same scene together and pushing other scenes apart. It is now a standard way to pre-train models on unlabeled satellite imagery. It also has a hidden dial, the temperature $\tau$, which controls how sharply the loss focuses on the hardest "negative" examples, and results are known to swing widely with it. Geographic deployment adds a second difficulty: a model trained on imagery from Europe and North America is used in Africa or Oceania, where nobody has labels with which to check which setting of the dial works. We propose to measure how large this problem is and then to address it with a selection rule that needs no labels. The rule uses two things every new region does have: the coordinates of its unlabeled tiles, and repeated images of the same places taken at different times.
The core idea is easy to state. A good representation of a new region should treat two images of the same place on different dates as near-identical, even though season, sun angle and sensor differ between them. It should also still separate neighbouring places that differ on the ground. A representation that has memorized local "style" (soil color, haze, viewing angle) fails the first test. A representation that has collapsed fails the second. We score every candidate pre-training configuration on these two properties using only unlabeled target-region imagery, and we test whether the score ranks configurations by their true out-of-region accuracy better than the standard practice of ranking by in-distribution validation accuracy.
Core Claim
We make two linked claims, and each can be falsified on its own. The first is empirical. On a geographic hold-out, out-of-region accuracy as a function of temperature and of the rule for defining positive pairs peaks somewhere different from in-distribution accuracy. Picking the in-distribution winner therefore costs measurable out-of-region accuracy, more than seed-to-seed noise. The second is methodological. A revisit-versus-neighbour score computed on unlabeled target tiles recovers most of that lost accuracy. Its key ingredient is the same-place, different-date term: plain spatial autocorrelation of embeddings should not work, because local style memorization also produces spatially smooth embeddings.
Where This Sits
The motivation comes from two places. WILDS and its unsupervised-adaptation extension benchmark many methods on FMoW-WILDS and PovertyMap-WILDS. They find that invariance methods rarely beat plain supervised training, and they name a "hyperparameter trap": adaptation methods look good only when tuned on target labels that would not exist in deployment. Neither paper studies contrastive temperature or proposes a label-free way out. Temperature Schedules for long-tail data shows that one fixed temperature serves dense and sparse parts of the data differently, and that cycling it between 0.1 and 1.0 helps. Temperature as Uncertainty makes the temperature a learned function of the input. Both suggest that the dial matters most exactly where data density is uneven, and satellite archives are extremely uneven across the globe. Neither was tested under geographic shift.
The closest geospatial contrastive work supplies the methods we sweep over, and none of it addresses selection. CSP aligns images with a learned encoder of latitude and longitude using a single global contrastive setup, and does not evaluate regions with zero coordinate overlap. Spatial Contrastive Learning for Remote Sensing treats nearby patches as positives under a fixed, hand-chosen distance threshold, and this threshold becomes one of our swept hyperparameters. FairDCL adds an adversarial urban/rural penalty to MoCo-v2 and keeps the default hyperparameters. The obvious combination at this intersection would be to make temperature a learned function of location. We deliberately do not propose that as the main contribution. A skeptic could reasonably expect the existing input-dependent temperature or a cosine schedule to capture most of that gain without geography, so we include both as competitors instead.
Research Program
We first build a controlled sweep. We pre-train ResNet-50 encoders with MoCo-v2-style contrastive learning on the unlabeled FMoW imagery curated in WILDS 2.0, always holding out one continental region entirely during pre-training. The sweep crosses five static temperatures (0.05, 0.1, 0.2, 0.5, 1.0) with four positive-pair rules: augmentation-only, and spatial-neighbour positives at three radii in the style of Bjorck et al. On top of these, we train the cosine temperature schedule and the learned input-dependent temperature, each with three seeds. To keep compute bounded, all runs use a reduced and identical epoch budget. Each encoder is then evaluated by a linear probe trained on labeled source regions, and we record accuracy on the held-out region. That accuracy is the "oracle" quantity a deployed practitioner could never see.
Next we define the label-free score. Take the unlabeled tiles of the target region. Let $s_{\text{rev}}$ be the mean cosine similarity between embeddings of the same location at different acquisition dates (FMoW contains such revisits). Let $s_{\text{nbr}}(h)$ be the mean similarity between distinct locations at geographic distance about $h$. We score a configuration by
$$S = s_{\text{rev}} - \max_{h \le h_0} s_{\text{nbr}}(h),$$
where $h_0$ is a short distance, a few kilometres. A high $S$ means the encoder ignores acquisition changes but still tells nearby places apart. To prevent leakage, we never use revisit pairs as training positives in the sweep. Revisits stay a held-out diagnostic, so the score is not simply rewarding the training objective.
Finally we analyze the results. For each held-out region, we rank all configurations by each selection rule. We report the Spearman correlation with oracle out-of-region accuracy and the regret: oracle-best accuracy minus the accuracy of the configuration the rule picks. Regret is the quantity a practitioner actually cares about. Repeating this leave-one-region-out over all FMoW regions gives a distribution of regrets rather than a single anecdote.
Baselines
The naive reference point is to use library-default settings (temperature 0.2, augmentation-only positives) with no selection at all. This shows what a typical practitioner gets today. Random selection from the sweep gives the chance-level regret.
The strongest existing competitor for selection is the protocol WILDS itself uses: choose hyperparameters by accuracy on a labeled in-distribution validation split. It is strongest because it is the accepted standard, and because the WILDS papers report that in-distribution and out-of-distribution performance are often correlated on these datasets. If that correlation holds across our sweep, our score has nothing to fix. Every rule chooses from the identical pool of trained encoders, so data, compute and the number of candidates are matched exactly. We will count the proposal as beating this baseline only if its median regret across held-out regions is lower by more than the seed standard deviation, and if its rank correlation with oracle accuracy is higher. We also compare against two other label-free signals computed on target tiles: the contrastive loss itself, and the standard alignment-and-uniformity statistics of the embeddings.
The strongest competitors for removing the need to select are the cosine temperature schedule of Temperature Schedules and the learned temperature of Temperature as Uncertainty. Each is run once at its published defaults, with the same epochs and seeds. If either matches the oracle-selected static configuration on the held-out region, selection is unnecessary, and we will report that.
The key ablation drops the revisit term and scores encoders by spatial smoothness alone, $-s_{\text{nbr}}$ over a range of distances. Our hypothesis predicts this ablation selects style-memorizing encoders and has higher regret than the full score. A second ablation drops the neighbour term, to check whether $s_{\text{rev}}$ alone rewards collapsed encoders.
We found no reported FMoW-WILDS accuracy number on Lacuna, so the downstream reference is our own rerun of supervised training under the WILDS protocol. For PovertyMap-WILDS, the published reference is the ERM Pearson correlation dropping from 0.60 in-distribution to 0.45 out-of-distribution (WILDS). A selected contrastive encoder that closes part of that gap under linear probing would be meaningful.
Why not just…?
Why not just select on in-distribution validation accuracy, as WILDS does? This is the most serious threat, and the first experiment exists to test it. If the in-distribution winner is also the out-of-region winner within seed noise, the hyperparameter trap does not apply to contrastive pre-training on these data. We would publish the sensitivity map as a negative result and drop the selection score. The reason to expect otherwise is the long-tail temperature evidence: low temperature favours fine distinctions within dense data and high temperature favours coarse grouping. Held-out regions are exactly the sparse, differently structured part of the distribution, so the best setting plausibly differs there.
Why not just use a temperature schedule or a learned temperature and stop tuning? These may genuinely flatten the sensitivity curve, which is why they are first-class baselines rather than footnotes. Two things limit them as a complete answer. Both were validated only on CIFAR-style benchmarks, and both leave the spatial positive radius untouched, which is a second dial with its own sensitivity. If they match oracle selection on the held-out regions, that is a useful finding in its own right. Our score would then be needed only for the radius, and we would say so.
Why not just rebalance sampling so sparse regions count more? Region-balanced sampling targets uneven data density directly and is cheap, so we add it as a fifth axis of the sweep instead of arguing against it. It does not remove the need to choose temperature and radius, and it does not tell a practitioner which resulting encoder to deploy. The selection problem therefore remains whether or not rebalancing helps.
Why not just fine-tune a geospatial foundation model with LoRA? Fine-tune Smarter, Not Harder shows that LoRA beats full fine-tuning on the Sen1Floods11 geographic hold-out (87.57% versus 82.07% mIoU). However, LoRA still has a learning rate and rank to choose, and that paper notes it does not resolve extreme shifts where the pre-trained space lacks support. As a secondary experiment, we apply the same selection comparison to LoRA hyperparameters on the Sen1Floods11 hold-out. This tests whether the revisit score generalizes beyond contrastive temperature.
Decisive Checks
Three outcomes settle the proposal. First, the sensitivity gap: across leave-one-region-out FMoW splits, in-distribution selection must show regret larger than seed noise. If it does not, the method half is moot. Second, the selection result: the revisit-versus-neighbour score must achieve lower median regret than in-distribution selection and than the loss and uniformity signals, and it must also do so on PovertyMap-WILDS. Because revisits are scarcer there, we report the neighbour-only and available-revisit variants separately rather than assume they exist. Third, the mechanism: the smoothness-only ablation must do worse than the full score. Otherwise our explanation, that the revisit term guards against memorized local style, is wrong, even if the score happens to help.
References
- WILDS: A Benchmark of in-the-Wild Distribution Shifts
- Extending the WILDS Benchmark for Unsupervised Adaptation
- Temperature Schedules for self-supervised contrastive methods on long-tail data
- Temperature as Uncertainty in Contrastive Learning
- CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations
- Accelerating Ecological Sciences from Above: Spatial Contrastive Learning for Remote Sensing
- Fair contrastive pre-training for geographic images
- Fine-Tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Models