Satellite foundation models are large networks pretrained, without labels, on huge archives of Sentinel and Landsat imagery. They are now routinely praised for "geographic generalization." Fine-tuned on a handful of labeled chips, they hold up on regions withheld from fine-tuning, while small networks trained from scratch collapse. These satellites image the whole planet on fixed orbits, so the unlabeled pretraining archive almost certainly contains the very places, and often the very scenes, that the downstream benchmark calls "held out." A geographic holdout split guarantees only that the labels of a region were unseen. It says nothing about whether the backbone memorized that region's terrain, soils and buildings during pretraining.
This note proposes a controlled study that measures how much of the reported small-data geographic generalization is caused by pretraining exposure to the evaluation footprint. The study builds two matched pretraining runs that differ only in whether the evaluation regions are present. It then asks whether the advantage of foundation models, and of parameter-efficient fine-tuning in particular, shrinks once exposure is removed. Our hypothesis is that the inflation is largest exactly where the field celebrates foundation models most, at tiny label budgets. A secondary product is a power-calibrated audit for checkpoints whose pretraining data is undisclosed. It uses the model's own masked-reconstruction error as the memory signal, and same-place and different-place imagery as clean controls.
Core Claim
The geographic benchmarks for satellite foundation models confound two different things. One is transfer to genuinely new terrain. The other is recall of terrain seen during unlabeled pretraining. We claim this confound is measurable, that it depends on how many labels are used for fine-tuning, and that it explains part of a known result. The candidate result is that low-rank adaptation (LoRA, which trains small added matrices and leaves the backbone frozen) beat full fine-tuning on the Sen1Floods11 geographic holdout. Fine-tune Smarter, Not Harder evaluated LoRA, prompt tuning and adapters on Prithvi 2.0 and Clay. They reported 87.57% mean intersection-over-union (mIoU) for LoRA against 82.07% for full fine-tuning on the Sen1Floods11 holdout, and attributed the gap to LoRA preserving general spatial features. An equally consistent explanation is that LoRA preserves memorized features of the holdout region itself. Our design separates these two explanations.
The precise quantity we estimate is an exposure gap. Let $\mathrm{mIoU}_{\text{full}}(b)$ and $\mathrm{mIoU}_{\text{excl}}(b)$ be holdout-region scores after fine-tuning with label budget $b$. The first comes from a backbone pretrained on the full corpus, the second from an otherwise identical backbone whose corpus excludes the evaluation footprints. We define
$$\Delta(b) = \mathrm{mIoU}_{\text{full}}(b) - \mathrm{mIoU}_{\text{excl}}(b).$$
The headline prediction is that $\Delta(b)$ is positive and decreases as $b$ grows. The reasoning is that a model with few labels leans hardest on what pretraining stored, while a model with many labels can relearn the region from supervision.
Where This Sits
The motivating evidence comes from three geospatial benchmarking efforts, none of which checks pretraining exposure. PANGAEA evaluates nine foundation models on eleven datasets across five continents. It finds that UNets trained from scratch lead on 6 of 11 datasets with full labels but fall behind models such as CROMA and SpectralGPT at 10% labels. GEO-Bench and GEO-Bench-2 standardize spatially held-out tasks and report large fine-tuning gains. Both treat a geographic split as sufficient evidence of out-of-distribution testing. We keep their tasks and splits and add the missing variable, which is whether the backbone saw the test footprint.
The geospatial evaluation literature constrains our design. Model Evaluation for Geospatial Problems shows that spatial autocorrelation leaks information across nearby train and validation points. It proposes buffered cross-validation with importance weighting, and warns that buffers waste scarce samples on small datasets. Its buffers act only at fine-tuning time. We move the buffer upstream into the pretraining corpus, where it costs no labeled data at all. That paper also tells us that exclusion zones must extend beyond the autocorrelation range, or the "excluded" model still sees near-duplicate neighbors. Position: Mission Critical argues that satellite data breaks the independent-sample assumption and reports drops of up to 30% on unseen disaster sites. We read that as evidence that truly unseen terrain is hard, which makes an inflated holdout score plausible.
The method for auditing is borrowed from the language-model contamination literature. When Is Benchmark Contamination Detectable? models a contaminated benchmark as a mixture of seen and unseen items. It shows detectability scales with $\alpha\rho\sqrt{m}$, where $\alpha$ is the contaminated fraction, $\rho$ the behavioral gap between seen and unseen items, and $m$ the audit size. Textbook Gaussian sample-size formulas failed in 9 of 9 cases below $m=50$. Their simulation-based planner reached empirical power 0.785 to 0.92 against a 0.80 target. They also found a hierarchy of signal strength: verbatim exposure leaves the strongest trace, paraphrase less, and answer-only exposure leaves almost nothing. Their framework assumes matched clean controls, which they note are undefined for continuous rasters. Our second contribution is to define such controls for satellite imagery.
Satellite pretraining is the image analogue of input-only exposure: the model sees the picture but never the label. This is a case their hierarchy does not cover and that we test directly. Language-side work on post-cutoff evaluation, such as LiveBench and Unveiling Causal Reasoning in LLMs, motivates one of our control arms.
The small-data literature explains why the label budget is the axis that matters. No Data Augmentation? shows that small validation sets cannot guide model selection, which they call a "validation crisis." Holdout regions in Sen1Floods11, built from only 446 hand-labeled chips worldwide, are tiny in exactly this sense. We therefore report region-bootstrap intervals rather than single scores. Trusting Small Training Dataset for Supervised Change Detection uses maximum logits as a scene-level reliability score that triggers a fallback to unsupervised detection. On Dubai, the reliability score of 0 matched a Kappa of 0.06, and the fallback reached 0.18. We test whether pretraining exposure silently inflates that kind of confidence signal.
Research Program
The core artifact is a pair of masked-autoencoder backbones of ViT-B size, trained with identical recipes, data volume, and compute on a Sentinel-1/Sentinel-2 corpus. A masked autoencoder is a network trained to fill in hidden patches of an image. The two runs differ in one respect. The "excluded" corpus removes every acquisition whose footprint falls within a buffer of any evaluation chip, across all dates. The removed tiles are replaced by tiles drawn from matched ecoregions elsewhere, so the total volume of data is unchanged.
The evaluation chips are the Sen1Floods11 holdout chips plus the PANGAEA and GEO-Bench-2 tasks that share these sensors. Following the autocorrelation logic of the geospatial evaluation paper above, we choose the buffer by estimating the distance at which feature correlation decays. We then report results at that distance and at double it.
A third "time-excluded" run keeps the evaluation locations but removes acquisitions within a window around each evaluation date. This separates memory of a place from memory of a scene. The distinction matters operationally. A flood model is deployed on known places at new times, so place exposure may be legitimate while scene exposure is not.
On top of the excluded backbone we run dose-response injections, modeled on the causal injections of the contamination paper. We continue pretraining briefly, under matched compute, on three kinds of data. The first is the verbatim evaluation scenes. The second is the same locations on other dates. The third is unrelated tiles as a drift control. This gives a curve of the exposure gap as a function of exposure type and amount, rather than a single comparison.
Each backbone is fine-tuned on each task at label budgets of 1%, 5%, 10% and 100%. We use three adaptation modes, namely linear probing, LoRA, and full fine-tuning, run in TerraTorch with the same hyperparameter search budget for every cell.
The audit component uses each backbone's masked-reconstruction error on a chip as its memory signal, the image counterpart of a language model's per-token surprise. For each evaluation chip we construct two clean controls. A temporal control is the same location on a date guaranteed to be absent from pretraining. A spatial control is an ecoregion- and biome-matched location beyond the buffer. Because we know the ground truth in our own runs, we can calibrate the two-stage resampling planner from the contamination paper on these controls, using regional audits of only dozens of chips. We then apply the calibrated audit to public checkpoints such as Prithvi 2.0 and Clay, whose exact pretraining footprints we cannot verify.
Baselines
The naive reference point is the score of a public checkpoint on the standard geographic holdout, as benchmarks report it today. The specific comparison is Prithvi 2.0 300M with LoRA, which Fine-tune Smarter, Not Harder reports at 87.57% mIoU on the Sen1Floods11 holdout, against 82.07% for full fine-tuning. We reproduce these numbers with their released configuration before running anything else, so that every later comparison starts from a verified anchor. This is also the strongest existing method on the question. It is the result that most directly claims small-data geographic generalization, and it is the one our hypothesis predicts is partly inflated.
The contamination-free floor is a UNet trained from scratch on the same label budgets and splits. Because it has no pretraining, its score cannot reflect exposure. PANGAEA shows UNets are competitive at full labels and weak at 10%. However, the summary we have records no per-dataset number, so we report no reported result found on Lacuna for the PANGAEA UNet at 10% labels and measure it ourselves. The original Sen1Floods11 paper gives older reference points for that dataset: 0.3389 flood-water mIoU for a ResNet50 network trained on weak labels, and 0.4571 for Otsu thresholding on permanent water.
The ablation that isolates the key ingredient is our matched pair, full versus excluded pretraining, under identical recipes, data volume, compute and tuning budgets. The proposal "wins" its central claim if the region-bootstrap 95% interval on $\Delta(b)$ excludes zero at the 1% or 5% budget. It wins its secondary claim if the LoRA-over-full-fine-tuning margin measured on the full backbone shrinks significantly on the excluded backbone. If $\Delta(b)$ is indistinguishable from zero at every budget, that is a clean negative result. It would mean current holdout scores can be trusted, and we would report it as such.
For the audit, the baselines are three. The first is the metadata check: intersect test footprints with the published corpus manifest. The second is a plain two-sample z-test on reconstruction error, sized with the Gaussian formula. The third is the calibrated planner. Beating the z-test means reaching nominal power near 0.80 at audit sizes below 50 chips, where the language-model results predict the Gaussian sizing fails. The planner's reported power range of 0.785 to 0.92 is the target.
Why not just…?
Why not just intersect test coordinates with the pretraining manifest? For any disclosed global corpus, this check will almost certainly say "exposed." That is useful but answers the wrong question. It tells us exposure happened, not whether exposure changed the score. Our matched pair measures the effect, which metadata cannot. The behavioral audit is reserved for checkpoints without a usable manifest. We concede that when a manifest exists, metadata is the right exposure test, and we use it to validate the audit on our own runs.
Why not just evaluate on imagery acquired after the pretraining cutoff? This is the most serious alternative, borrowed from post-cutoff language benchmarks, and it is cheap. Its weakness is that orbits revisit the same places. Post-cutoff imagery removes scene exposure but keeps place exposure, and newly labeled post-cutoff chips are scarce. Our time-excluded run is exactly this design, performed in a controlled way. If it recovers most of the full-versus-excluded gap, then post-cutoff evaluation is sufficient, and we will recommend it as the cheap standard over our heavier protocol.
Why not just use buffered spatial cross-validation during fine-tuning? Buffered cross-validation, as formalized in Model Evaluation for Geospatial Problems, controls leakage among labeled samples. It cannot touch exposure that occurred in an unlabeled archive before any fold existed, and that paper itself notes buffers are costly on small datasets. We include buffered fine-tuning splits in every condition. The exposure gap is measured on top of them.
Why not just train from scratch and avoid the issue? The UNet floor does this. PANGAEA shows from-scratch models lose badly at small budgets. If the excluded backbone still clearly beats the UNet, then pretraining offers real transfer even after exposure is removed, and from-scratch training would throw that transfer away.
Proposed Experiments
The first decisive check is the shape of $\Delta(b)$ across budgets and adaptation modes on the Sen1Floods11 holdout and the matching PANGAEA and GEO-Bench-2 tasks. Uncertainty comes from resampling whole regions, not individual chips, because chips within a region are correlated. Our prediction is that the gap is largest at 1% labels and under LoRA or linear probing, and near zero for full fine-tuning at 100%.
The second is the injection curve. We expect verbatim scenes to produce a larger gap than same-place-other-date exposure, which in turn should exceed drift. This would extend the exposure hierarchy reported for language to input-only image exposure. If same-place exposure alone reproduces most of the gap, the field's notion of a "held-out region" needs to change, not just its handling of dates.
The third check concerns confidence signals. We compute the logit-based scene reliability score of Trusting Small Training Dataset on full and excluded backbones. We then test whether exposure raises confidence on regions where accuracy is poor, which would blind the fallback mechanism. The score's agreement with Kappa on the excluded backbone serves as the reference.
The fourth is the audit's power curve. We plot detection power against audit size and exposure type on our own runs, where ground truth is known, comparing the Gaussian-sized test with the calibrated planner. We then apply the calibrated audit to Prithvi 2.0 and Clay on the Sen1Floods11 holdout, and report either a detection or an explicitly power-qualified non-detection.
Together these checks tell the community whether "geographic holdout" scores for satellite foundation models are measuring transfer, and which cheap evaluation fix is enough.
References
- Fine-Tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Models
- PANGAEA: A Global and Inclusive Benchmark for Geospatial Foundation Models
- GEO-Bench: Toward Foundation Models for Earth Monitoring
- GEO-Bench-2: From Performance to Capability, Rethinking Evaluation in Geospatial AI
- Model Evaluation for Geospatial Problems
- Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning
- LiveBench: A Challenging, Contamination-Limited LLM Benchmark
- Unveiling Causal Reasoning in Large Language Models: Reality or Mirage?
- No Data Augmentation? Alternative Regularizations for Effective Training on Small Datasets
- Trusting small training dataset for supervised change detection
- Sen1Floods11: a georeferenced dataset to train and test deep learning flood algorithms for Sentinel-1