When a lake shrinks, a wetland browns, or a forest thins, scientists usually want to know how much of the change came from regional climate and how much from local human action such as irrigation, logging, or pond abandonment. The standard method is to compare the managed area with a nearby protected reserve that shares the same climate. Whatever changed in the reserve is counted as climate, and the remainder is counted as human impact. Two problems limit this. Suitable reserves often do not exist, and a satellite image of a drought-browned field and an abandoned field can look the same. We propose to learn the reserve instead of finding one. The model is trained, without labels, to predict how a location's learned image representation should change using only what happened in the surrounding region, while the location's own immediate neighborhood is hidden from it. The gap between the predicted change and the observed change is a candidate map of local, non-climatic change. A small number of expert labels are then used only to calibrate and name that gap. They are not used to teach the model what change looks like.
This proposal places four lines of work in specific roles. Latent feature prediction is the core mechanism. It refers to self-supervised models that learn by predicting the internal representation of a hidden part of the input rather than its raw pixels. Driver attribution is the scientific question. Label efficiency is the operating constraint. The subjectivity of manual geological and environmental interpretation shapes the evaluation, because attribution labels drawn by experts disagree, and that disagreement sets the ceiling any model can honestly be scored against.
Core Claim
The claim is that a masked, context-only predictor operating in a pretrained Earth-observation embedding space can serve as a learned counterfactual control landscape. The predictor sees how the surrounding region's embeddings changed, sees regional climate covariates, and sees the location's static terrain and soil. From these it predicts the location's own change. The residual it leaves isolates localized disturbance more reliably than two alternatives: spectral-index high-pass filtering, and supervised change classifiers trained on the same small label budget. The advantage should be largest where the same regional climate shock produces different responses on different terrain.
Formally, let $z_{x,t}$ be the frozen encoder embedding of location $x$ at time $t$, and let $\Delta z_x = z_{x,t_2} - z_{x,t_1}$. A predictor $g$ receives the embedding changes $\{\Delta z_u\}$ of locations $u$ lying in an annulus between radii $r$ and $R$ around $x$, regional climate covariates $c$, and static covariates $s_x$ (elevation, slope, soil). It is trained on all locations to minimize
$$\mathcal{L} = \mathbb{E}_x \, \big\| \Delta z_x - g(\{\Delta z_u\}_{r<|u-x|<R},\, c,\, s_x) \big\|^2 ,$$
and the attribution score is the residual $e_x = \Delta z_x - g(\cdot)$. The inner exclusion radius $r$ does the work that a physical control landscape normally does. A disturbance confined to within $r$ of $x$ cannot leak into the prediction. Because most of the landscape is not locally disturbed in any given interval, $g$ learns the regionally driven part of change, and $e_x$ carries what the region cannot explain.
The key assumption, stated plainly, is that climate forcing is spatially smooth at scale $r$ even when the response to it is not. The static covariates are meant to absorb heterogeneous responses, such as basins of different shape or hydrogeology reacting differently to the same drying. The proposal fails where climate forcing itself is sharper than $r$, as with convective storms or local fog belts. We will measure this failure case directly rather than assume it away.
Relation to Prior Work
The mechanism comes from AnySat. AnySat pretrains a multi-sensor Earth-observation encoder by predicting latent representations of masked patches with a joint-embedding predictive architecture, plus a contrastive term. It reports "+10.2 mIoU" on BraDD-S1TS deforestation and "+11.0% accuracy" on SICKLE crop mapping over specialized architectures. AnySat evaluates only static discriminative tasks. It never asks why a surface changed, and its masking is spatial within a single snapshot. We reuse its encoder frozen and move the prediction problem to change across time and space, with an explicit exclusion zone.
A remote-sensing foundation-model survey (Foundation Models for Remote Sensing and Earth Observation: A Survey) motivates the gap: current models "treat images as static snapshots" and "struggle with complex reasoning (e.g., distinguishing between flood and fire damage)." The case for predicting in latent space rather than pixel space (Self-Supervised Learning via Latent Feature Prediction) is that pixel reconstruction spends capacity on haze, clouds, and illumination. That matters more for a residual than for a classifier, because every unmodeled nuisance shows up as false "human" signal.
On the attribution side, existing methods motivate and constrain the design. The protected-reserve subtraction method and spatial-autocorrelation fingerprinting (climate diffuse and smooth, human effects local and sharp) are the scientific precedent. Our model is essentially a learned, embedding-space version of that fingerprinting idea, and reserve subtraction becomes one of our validation tests rather than a requirement. Physically based pipelines such as the RWEQ wind-erosion attribution on Google Earth Engine for the Aral Sea (Attribution and Risk Assessment of Wind Erosion in the Aral Sea Regions Using Multi-Source Remote Sensing and RWEQ on GEE) and drought-propagation lag analysis in the Haihe basin (Characteristics and Lag Time of Meteorological Drought Propagation to Hydrological Drought in the Haihe River Basin) are interpretable. However, they rely on index thresholds and idealized catchments. We use them as sources of study regions and as comparison attributions, not as competitors on the same labels. Field studies such as the Sanjiang plain disturbance-gradient work (Differences in functional traits of herbaceous plants in Sanjiang plain wetland under human disturbance gradient and their response strategies to environmental changes) supply an independently measured disturbance gradient around the Honghe reserve against which residuals can be checked. The US lake study (Abrupt changes in algal biomass of thousands of US lakes are related to climate and are more likely in low-disturbance watersheds) is a caution. Climate-driven abrupt change can be concentrated in low-disturbance places, so "abrupt" must not be read as "human," and our score deliberately depends on spatial locality, not temporal abruptness.
The interpretation-bottleneck literature constrains the evaluation. ConeQuest shows supervised segmentation falling from 52.52% to 15.04% IoU on unseen Martian regions. Landslide4Sense shows that even full supervision (ResU-Net, F1 71.65%) confuses fresh scarps with bare soil or clearings. Both warn that a supervised attribution classifier trained on a few regions will learn regional texture, not driver. Human–machine collaboration work (A human-machine collaborative approach measures economic development using satellite imagery) supplies the workflow we adopt: the model ranks locations and experts review the most uncertain ones.
Research Program
We build three components.
The first is the residual predictor. It is a small transformer that takes the annulus of neighbor embedding changes as a set of tokens, with relative positions, plus climate and static covariates. It sits on top of a frozen AnySat-style encoder applied to Sentinel-1/2 time series. Training is fully self-supervised over large unlabeled regions, including areas outside Europe and North America, to partly offset the geographic bias of the pretraining data.
The second is a calibration head. It maps the residual vector $e_x$ to a small set of expert-named driver categories, such as clearing, water diversion, aquaculture change, or none. It is fit with 1%, 5%, and 10% label budgets, so the residual does the representation work and labels only name it.
The third is an agreement protocol for the subjectivity problem. For a held-out set, three or more experts independently label each location's dominant driver. We report pairwise expert agreement and score the model against each expert on the same scale. "Human-level" then means falling within the inter-expert band, not matching one annotator's opinion.
Study regions are chosen where attribution is contested and documented in the file. These are the Aral Sea basin, the Haihe basin, the Sanjiang plain around the Honghe reserve, and the Amazon region covered by BraDD-S1TS.
Baselines
The naive reference point is raw embedding change magnitude, $\|\Delta z_x\|$, with no prediction. It flags every change, climate or human. If our residual does not clearly beat it at separating local disturbance from region-wide change, the predictor adds nothing.
The classical competitor is spectral fingerprinting with a spatial high-pass filter. We compute the change in standard indices (vegetation, water, and bare-soil indices), subtract the mean change in the same annulus, and optionally regress out climate covariates. This is the learned method's closest non-learned relative and the most dangerous baseline. It is run on the same imagery, the same annulus radii, and the same covariates. Its regression is fit on the same unlabeled data, so it receives identical inputs.
The strongest existing learned competitor is an AnySat encoder with a supervised change head: frozen embeddings at $t_1$ and $t_2$, concatenated, with a linear probe and a small MLP probe trained on the driver labels at each budget. This is AnySat's own evaluation style, and it gets the same labels and the same hyperparameter search budget, with 20 configurations on a validation split per budget. For BraDD-S1TS, AnySat reports a "+10.2 mIoU" gain over specialized architectures. The file gives only this relative gain, not an absolute mIoU for AnySat's linear probe, so our reference is the number we reproduce ourselves with the released model under AnySat's protocol. For driver attribution as such, no reported result was found on Lacuna for any learned method.
The protected-reserve subtraction baseline applies wherever a reserve exists, as around Honghe. There we compare our residual to observed-minus-reserve change directly.
Three ablations isolate the key ingredients. The first sets the exclusion radius $r$ to zero, so the predictor can see the location's neighborhood; this should collapse the residual toward noise. The second removes the static covariates, which tests the heterogeneous-response claim. The third runs the same predictor in pixel or index space instead of embedding space, and separately with a reconstruction-pretrained encoder, which tests whether latent prediction matters.
Our method beats the field if its calibrated residual exceeds the supervised AnySat probe in balanced accuracy across held-out regions at the 1% and 5% budgets. It must also beat the high-pass index baseline in the stratum where regional climate shocks are strong. A win only on easy, drought-free intervals would not count.
Why not just…?
Why not just high-pass filter a vegetation or water index? This is a real threat, and it may match us wherever land cover is uniform and the climate response is homogeneous. The argument against it is specific. The same drying produces different index changes on different terrain and soils, a point made by Liu and colleagues on lake desiccation. Index spaces also carry haze and phenology artifacts that a subtraction does not remove. The ablation comparing embedding space with index space, stratified by terrain heterogeneity within the annulus, settles this. If the filter ties us in heterogeneous strata, the embedding predictor is not justified, and we would report that and fall back to the filter plus learned calibration.
Why not just train a supervised change classifier on AnySat features with the few labels available? It is the most natural move, but with a few labels from a few regions it will learn which textures co-occur with "human" in those regions. ConeQuest's collapse from 52.52% to 15.04% IoU on unseen regions shows the scale of this risk in geoscience imagery. Our leave-one-region-out protocol measures exactly this. The classifier may win in-region. The claim is that the residual transfers, and if it does not, the claim is false.
Why not just use reserves or a Budyko water balance? Where a matched reserve exists, reserve subtraction is the right answer, and we treat it as ground truth rather than a rival. The problem is that such reserves are usually missing, as in the Aral and Haihe basins. Budyko-type balances attribute catchment-level runoff, not pixel-level land-surface change, so they cannot say which fields or ponds drove the change. We report agreement with Budyko-style catchment totals as a consistency check.
Why not just collect more expert labels? Attribution labels disagree between experts, and more labels from the same subjective process raise the cost without fixing the ceiling. The agreement protocol quantifies how much additional labeling could plausibly help.
Proposed Experiments
The first decisive check is controlled injection. We take real Sentinel time series from drought and non-drought years in the Haihe and Aral regions and splice in real disturbance patches, such as clearings and new ponds, taken from other places and times at known locations. We then measure detection AUC of the residual as a function of drought intensity. A method that separates drivers should keep its AUC as the regional shock grows. The raw-change and index baselines should lose AUC, because climate change fills their maps.
The second check uses the reserve. Around Honghe, residuals inside the reserve should be near zero on average. Residuals outside it should rise along the disturbance gradient documented by the Sanjiang field study, and should correlate with reserve-subtracted change.
The third check is a zero-shot and few-label test on BraDD-S1TS. There, mapped deforestation is human-driven by construction. We score the uncalibrated residual as a zero-shot detector, then add the calibration head at 1%, 5%, and 10% of labels, and compare against the reproduced AnySat probe at the same budgets. Sen1Floods11 serves as a negative control. Large-area flooding should produce low residual relative to raw change, while small-scale flooding should not, which directly probes the smoothness assumption.
The fourth check is multi-region attribution under expert disagreement. Leave-one-region-out balanced accuracy is reported against each expert and against the inter-expert band.
The sweep over the exclusion radius $r$ is reported throughout. It shows at what spatial scale local and regional change separate, and where the method stops working.
References
- AnySat: One Earth Observation Model for Many Resolutions, Scales, and Modalities
- Foundation Models for Remote Sensing and Earth Observation: A survey
- Revisiting Feature Prediction for Learning Visual Representations from Video
- Attribution and Risk Assessment of Wind Erosion in the Aral Sea Regions Using Multi-Source Remote Sensing and RWEQ on GEE
- Characteristics and Lag Time of Meteorological Drought Propagation to Hydrological Drought in the Haihe River Basin
- Differences in functional traits of herbaceous plants in Sanjiang plain wetland under human disturbance gradient and their response strategies to environmental changes
- Abrupt changes in algal biomass of thousands of US lakes are related to climate and are more likely in low-disturbance watersheds
- ConeQuest: A Benchmark for Cone Segmentation on Mars
- Landslide4Sense: Reference Benchmark Data and Deep Learning Models for Landslide Detection
- A human-machine collaborative approach measures economic development using satellite imagery