Large models for satellite imagery are pre-trained on years of imagery from the same archives that downstream benchmarks are cut from. Nobody currently checks whether a model scores well on those benchmarks because it saw the test period during pre-training. The check borrowed from language models, a temporal cutoff (score the model only on data dated after its training data ends, and look for a drop), works badly here. The Earth's seasons repeat every year, so a model that has only memorized last year's greening curve still does well after the cutoff. Satellite data also has a second, quieter leak. Clouds leave gaps in the time series, and preprocessing pipelines routinely fill those gaps by interpolating between the observations on either side. Two-sided filling pulls future information into the past.
This note proposes a detector that does not depend on a cutoff. The core idea is to use events the model could not have inferred honestly. Grassland mowing is a human decision and cannot be predicted from weather. When a mowing event falls inside a cloud gap in the Sentinel-2 record but is visible to Landsat on the same days, a model that is fed only Sentinel-2 context has no legitimate way to place the event. If it recovers such hidden events more often than an exposure-free model can, it is reading leaked information. We calibrate this "canary" test by deliberately injecting test-period data into pre-training at known doses. We then show that the canary recovers the dose where the cutoff test and climatology-relative scores do not.
Core Claim
In cyclical, cloud-gapped satellite time series, a model's skill on unpredictable events hidden inside observation gaps is a calibrated, dose-sensitive measure of benchmark contamination. Post-cutoff decay is not, because seasonality masks it. The note makes two specific predictions. First, the canary recovery rate rises monotonically with the amount of injected test-period data, while post-cutoff decay stays flat or noisy. Second, two-sided gap filling in benchmark preprocessing inflates canary recovery even when there is no pre-training overlap at all. That second effect is a contamination channel created by preprocessing, which the literature has not measured.
Where This Sits
The motivating evidence comes from language-model evaluation. LiveCodeBench builds a benchmark of timestamped coding problems and reads contamination off a "contamination cliff" after a model's cutoff. Test of Time then shows that this temporal signal is fragile. Rephrasing the same theorems into multi-step problems erased the post-cutoff drop: Cloze recall fell by 3.83 points for GPT-4o-mini, while rephrased physics problems scored 2.19 points higher after the cutoff. Renaming variables in LiveCodeBench removed its decay as well. We take that negative result as a constraint. Any satellite detector built on cutoff decay inherits this fragility, and seasonality plays the same role here that rephrasing plays in text. That is why we score on events that do not recur.
The satellite side supplies both the data and the failure modes. Multi-Modal Learning for Geospatial Vegetation Forecasting introduces GreenEarthNet and Contextformer. Contextformer handles clouds by predicting change relative to the last cloud-free observation, which is a causal fill, and it uses "contamination" only in the sense of clouds. The mowing-detection work documents that interpolation across cloud gaps bridges over mowing dips. It also provides the Greek gold-standard parcels (Harmonized Landsat Sentinel, HLS, 2020) and the German pseudo-labelled pre-training set we reuse. SSL4Eco provides a global, phenology-sampled Sentinel-2 pre-training corpus and the SeCo-Eco contrastive recipe. Because that recipe samples every location across its seasonal cycle, a memorizing encoder would have exactly the "seasonal lookup table" that defeats cutoff tests. The aerosol study shows that gap-filled products carry atmospheric artifacts forward. That is a reminder that what fills a gap is never neutral.
None of these papers asks whether a geospatial model's benchmark score reflects exposure to the test period. TimeWak is the closest provenance tool. It watermarks synthetic time series by chaining hashes along time, but it assumes fixed-length, regularly sampled windows and works only on data the defender generates. It cannot audit a model trained on public archives.
Research Program
The program has three parts.
The first part builds the canary set. Within the Greek HLS parcels, we find mowing events whose drop in the Enhanced Vegetation Index (a greenness measure computed from red, near-infrared and blue reflectance) is observed by Landsat. On those same dates, Sentinel-2 is cloud-masked over a window of at least $k$ days on both sides, with $k$ swept from 10 to 40 days. We build matched non-canary controls: hidden spans of the same length and season in which no event occurred. These controls measure the false-alarm rate. The model receives only Sentinel-2 context outside the gap, plus meteorology, and must say whether an event happened inside the gap and when. The canary score is the timing-tolerant detection rate on canaries, minus the false-alarm rate on controls.
The second part runs a controlled injection. We pre-train SeCo-Eco-style encoders on SSL4Eco plus the German silver set. Into this corpus we inject Greek 2020 imagery at doses of 0%, 1%, 5% and 25% of canary parcels, in three forms. The first form is raw Sentinel-2 only, which leaves the gaps intact. The second adds Landsat for the gap dates, which is cross-sensor leakage. The third is Sentinel-2 gap-filled two-sidedly using Landsat-informed interpolation, which is preprocessing leakage. Every encoder is then fine-tuned on the same Greek training split. Separately, with no injection at all, we rebuild the benchmark's own context sequences under causal filling (last valid observation, as in Contextformer) versus two-sided filling. This isolates the leak that preprocessing alone creates.
The third part compares detectors. Each detector has to rank models by their true injected dose. This tells us which detectors actually measure contamination.
Baselines
The naive reference is the "do nothing" check that practitioners use today. It reports the benchmark score and trusts a geographic split, as the German-to-Greek transfer in the mowing paper does. It gets the same models, and success for our method means flagging contamination that this check cannot see by construction.
The strongest existing contamination detector is temporal-cutoff decay in the style of LiveCodeBench. Each encoder's injected data ends at a known date. We compare accuracy on test parcels before and after that date, using the same parcels, seasons and fine-tuning budget. No reported result found on Lacuna for cutoff decay applied to satellite models. Our method beats it if its rank correlation with true dose is significantly higher across the 12 dose-by-form conditions, with three seeds each.
The strongest domain-native signal is skill relative to climatology, meaning historical averages. Contextformer reports a 66.8% Outperformance Score against climatology on GreenEarthNet, with $R^2$ of 0.62 and RMSE of 0.14 at 100 days. One could argue that anomaly skill beyond climatology already exposes memorization. We compute it for every encoder on GreenEarthNet-style forecasting, with the same fine-tuning and tuning budget. Our method beats it if climatology-relative skill rises with dose at most as fast as legitimate weather-driven skill does, while the canary score separates the dose levels.
As a task-performance anchor, the distilled ResNet18 and Conv+Transformer models in the mowing paper reach 82.3% F1 with 84.4% recall on the Greek parcels. Our dose-0 models should land near this figure, which shows the pipeline is faithful. Any gain above it at higher doses is inflation from contamination.
Two ablations isolate the key ingredient. The first scores detection on all events rather than only gap-hidden canaries; if that works equally well, the hiding adds nothing. The second keeps the gaps but replaces mowing with gap-hidden seasonal transitions, such as greenup onset, which weather can predict; if that works equally well, unpredictability adds nothing.
Why not just…?
Why not just deduplicate the pre-training corpus against the benchmark by tile, location and date? When the corpus is fully known, deduplication is the right first step, and we use overlap counts as ground truth in the injection study. Deduplication has three blind spots, though. It misses cross-sensor exposure: HLS harmonizes Landsat and Sentinel-2, so a Landsat scene for the same parcel and day has a different tile ID. It misses the preprocessing leak, because a two-sided fill involves no pre-training overlap at all. And it cannot audit released checkpoints whose corpora are only loosely described. The injection results will show how much inflation survives perfect tile-level deduplication. If the answer is nearly none, this alternative wins, and we will report that.
Why not just use a temporal cutoff split? Seasons recur, so a memorizing model keeps its skill after the cutoff. Test of Time already shows that cutoff decay vanishes under mild transformation even in text. The dose-correlation comparison above tests this directly instead of assuming it.
Why not just score skill against climatology? It is the most serious competitor. Its weakness is that weather legitimately produces year-specific anomalies, so anomaly skill mixes honest physics with leakage. Mowing dates are set by farmers, so recovering them inside a gap cannot come from weather. If climatology-relative skill tracks the dose as well as the canary score does, we will drop the canary claim and recommend the simpler metric.
Proposed Experiments
The first decisive check is the dose-response curve. The canary score at dose 0 estimates the honest ceiling, which is what context and weather allow. It should rise with cross-sensor and two-sided-fill injection, and stay near that ceiling for raw Sentinel-2 injection where the gaps remain gaps. That last condition is the internal negative control. The second check is the preprocessing audit. With zero pre-training overlap, two-sided filling of the benchmark context should raise canary recovery and overall F1 compared with causal filling, and the size of this gap measures how much an evaluation convention alone inflates scores. The third check is detector calibration: rank correlation with true dose, and how many test parcels each detector needs to separate dose 0 from dose 5%. As an exploratory extension, we apply the canary test to publicly released remote-sensing backbones such as RSP and RVSA after fitting a temporal head. Since their exposure is unknown, results there are flags rather than verdicts.
The main risk is that canaries are rare. Long Sentinel-2 gaps that coincide with Landsat-observed mowing may yield only tens of events in the 2020 Greek data. We will report confidence intervals and extend the search to the German parcels, where clouds are more frequent, before drawing conclusions.
References
- LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code
- Test of Time: Rethinking Temporal Signal of Benchmark Contamination
- Multi-Modal Learning for Geospatial Vegetation Forecasting
- Large Scale Mowing Event Detection on Dense Time Series Data Using Deep Learning Methods and Knowledge Distillation
- SSL4Eco: A Global Seasonal Dataset for Geospatial Foundation Models in Ecology
- Monitoring the impact of aerosol contamination on the drought-induced decline of gross primary productivity
- TimeWak: Temporal Chained-Hashing Watermark for Time Series Data