Causally Filtered Activation Shifts for Unseen Geographies

Satellite models trained on some parts of the world often lose 20–30% accuracy when used on a continent or disaster site they never saw (Mission Critical – Satellite Data is a Distinct Modality in Machine Learning). Current fixes are costly in one way or another. Adapters like LoRA need labeled data from the new region. Test-time training, which updates the network on each new batch, needs backpropagation through the encoder at deployment. This proposal takes a cheaper route. It asks which directions inside a frozen geospatial foundation model encode "where this image came from" and are actually used by the prediction head. It then removes only those directions' regional offset at inference time, using a single forward pass over unlabeled images from the new region and no weight updates.

The central bet concerns selection, not steering. The model's internal activations carry a lot of regional information. Some of it is nuisance, such as sensor geometry, soil background and local haze. Some of it is signal: crops, biomass and building styles really do differ between regions. Some of it the decoder ignores entirely. The simplest test-time fix shifts all activations of the new region toward the training-region average, and that erases the signal along with the nuisance. We claim that a causal test run once on labeled source regions can tell these three kinds of direction apart. We further claim that shifting only the "used but label-irrelevant" directions closes more of the geographic gap than shifting everything. If uniform shifting works just as well, the causal machinery is unnecessary, and the experiments are designed to show that outcome plainly if it occurs.

Core Claim

Let $h \in \mathbb{R}^d$ be a patch-token activation at an intermediate layer of a frozen encoder. Let $P_N$ be the projection onto a "nuisance subspace" $N$. Let $\mu_S$ be the mean activation over training (source) regions, and $\hat\mu_T$ the mean over an unlabeled batch from the target region. The intervention is

$$h' = h - P_N(\hat\mu_T - \mu_S),$$

applied before the frozen decoder. Everything turns on how $N$ is chosen. Directions enter $N$ only if they pass two tests on source data. The first is decodability: a linear probe predicts the source region or biome from them. The second is causal use without label change: swapping the component along the direction between two same-label chips from different regions changes the decoder's output. A direction that is decodable but causally inert is left alone, because shifting it cannot help. A direction whose swap moves predictions only between different-label pairs is also left alone, because it carries semantics.

The prediction is specific. On tasks where the label distribution itself changes across geography, the causally filtered shift should beat both a uniform mean shift and a shift along probe-only (decodable) directions. On tasks where regional variation is mostly appearance, such as flood water, all three may perform similarly.

Why this is not already done

Three existing lines of work come close, and each misses a different piece. Mixstyle-Entropy frames domain generalization causally. It mixes high-entropy channel statistics during training and perturbs embeddings at test time to find unreliable samples, reaching 42.6% mIoU on GTA5→Cityscapes against 36.0% for plain training. It is a training-time heuristic on convolutional networks, it has never been applied to satellite data, and it does not test whether the channels it mixes are actually used. STAR detects spatial clusters of error and grows private regional layers, raising California land-cover weighted F1 from 0.49 to 0.67. It changes the network's structure, needs gridded regions, and cannot serve a region with no labels. MMEarth-Bench adapts to held-out Africa by backpropagating a multi-sensor reconstruction loss at test time. It never identifies which features cause the errors, and it pays for a backward pass at deployment.

On the intervention side, SteerVLM builds a small plug-in module that steers vision-language activations, but its steering signal comes from paired text prompts, which have no counterpart for soil reflectance or radar backscatter. The closest existing work is therefore either causal but not diagnostic (Mixstyle-Entropy), spatial but not activation-level (STAR), or adaptive but gradient-based (TTT-MMR). Our proposal is a label-free, gradient-free, per-direction causal selection of what to shift.

Two findings from interpretability both motivate the filter and constrain it. Analyzing (In)Abilities of SAEs via Formal Languages shows that features can match a concept perfectly while having zero causal effect. This is why decodability alone must not define $N$. The same paper finds that top-$k$ sparse autoencoders (networks that rewrite activations as a few active interpretable units) reconstruct far better than $L_1$ ones, at 92–99% versus under 33%, so we use top-$k$. Can SAEs reveal and mitigate racial biases of LLMs in healthcare? reports that ablating a single entangled latent cut stereotyped generation from 88% to 46% but had almost no effect on complex clinical decisions. This is a direct warning: region may be spread across many entangled directions that single-latent edits cannot reach. We address that by working with subspaces and by measuring how much of the gap each subspace size closes. Endogenous Resistance to Activation Steering shows that models can undo interventions downstream, so we measure whether a shift applied at layer $\ell$ survives to the decoder input.

Research Program

The work has three stages.

Stage 1: audit. We take frozen Prithvi 2.0 300M and Clay v1 with decoders trained on source regions only, following the TerraTorch setup of Fine-tune Smarter, Not Harder. At several depths we fit top-$k$ sparse autoencoders and also plain linear region probes, giving two candidate bases. For each candidate direction we run interchange interventions between source chips from different regions or events and record three things: decodability, the change in decoder output on same-label pairs, and the change on different-label pairs. This produces a map of region-encoding directions in geospatial foundation models. It also answers an open question by itself, namely what fraction of "geographic" directions are causally inert.

Stage 2: intervention. Directions that pass the filter form $N$. At deployment we compute $\hat\mu_T$ from one forward pass over a batch of unlabeled target chips and apply the shift above. Because Sen1Floods11 spans 11 flood events, we can choose $N$ by leave-one-source-event-out validation, so the held-out target is never used for selection.

Stage 3: shared intervention interface. All methods, including ours, uniform shifting, LoRA and TTT, plug into the same hook between encoder layers and decoder. This follows the decoupled-stage pattern of OmniEvaluator, which cut integration cost from $N \times M$ to $N + M$. The interface guarantees that every comparison uses identical data splits, backbones, decoders and evaluation code. It is infrastructure, not a claim.

Baselines

The naive reference point is the frozen source-trained model with no adaptation, evaluated directly on the held-out region. The strongest reported number in this setting comes from Fine-tune Smarter, Not Harder: LoRA on Prithvi 2.0 300M reaches 87.57% mIoU on the Sen1Floods11 geographic hold-out set, against 82.07% for full fine-tuning. That LoRA model is our starting point, and beating it means a statistically reliable mIoU gain on the same hold-out set with no target labels and no gradient steps. We will rerun it ourselves under our splits and seeds rather than rely on the reported figure.

The key ablation is uniform mean alignment: the same equation with $P_N$ replaced by the identity at the same layer. This is the classic test-time feature-statistics correction. Two further ablations isolate the causal filter. One uses the probe-only subspace, meaning decodable directions without the causal test, matched in dimension. The other uses a random subspace of the same dimension. Our method must beat the probe-only and uniform versions on at least the label-shifted tasks to justify itself.

TTT-MMR from MMEarth-Bench is the strongest label-free adaptive competitor on the Africa hold-out. We run it with its own recommended geographic batching under the same encoder. We report accuracy, per-chip latency and peak memory; no reported result found on Lacuna for its scalar scores. Matching its accuracy at a fraction of its deployment cost counts as a win.

A LoRA model tuned on labeled target data gives an oracle upper bound, so we can report the fraction of the gap each method closes. Mixstyle-Entropy's training-time mixing, ported to the ViT backbone, is a secondary baseline. Its reported numbers (88.5% on PACS, 42.6% on GTA5→Cityscapes) are not comparable to satellite tasks, and the port is our own. All tunable choices, including layer, subspace size and batch size for $\hat\mu_T$, get the same leave-one-source-region-out budget for every method.

Why not just…?

Why not just shift all activations toward the source mean? This is the main threat, and it is built in as the key ablation. On flood mapping, regional variation may be mostly nuisance, and uniform shifting could match our method there; we would report that result. The entanglement evidence from the healthcare SAE study and the spatial heterogeneity of crop signatures (Western Corn Belt trends) suggest a different outcome on m-SA Crop Type, m-Cashew and MMEarth biomass. On those tasks, uniform shifting should remove real regional signal and hurt. If it does not, the proposal reduces to a cheap diagnostic study.

Why not just fine-tune LoRA on the new region? LoRA works well when labels exist, and it defines our oracle. A newly flooded area or an unmapped continent has no labels at deployment time, and LoRA trained on source regions alone is precisely the model whose geographic gap we are trying to close.

Why not just use test-time training? TTT-MMR is effective on held-out Africa, but it backpropagates through a 300M-parameter encoder for every batch. Its authors also note that multi-sensor inputs can overfit regional correlations. The latency and memory measurements will show whether our forward-only shift gives up accuracy for cost, and by how much.

Why not just feed in coordinates? Fine-tune Smarter, Not Harder tested coordinate and timestamp inputs and found negligible gains. Our audit may help explain this: if the model already infers location from pixels, the useful move is to act on how that inferred location is used, not to add more of it.

Proposed Experiments and Decisive Checks

The benchmarks are the Sen1Floods11 geographic hold-out set and leave-one-event-out splits across its 11 events (Sen1Floods11), measured in mIoU. We add m-SA Crop Type and m-Cashew Plantation as label-shifted geographic transfers, and the MMEarth-Bench Africa hold-out for regression (biomass, soil pH, soil carbon) measured by RMSE. Every experiment is run on both Prithvi 2.0 and Clay v1.

Four results would settle the claim. First, the audit must show that a substantial share of region-decodable directions are causally inert. This would carry the "interpretability illusion" into geospatial models and justify the filter. Second, on the label-shifted tasks, the causally filtered shift must beat both uniform and probe-only shifts, and uniform shifting should underperform the unadapted model somewhere. Third, a shift applied at layer $\ell$ must still be present at the decoder input, measured as the remaining offset along $N$. If later layers undo it, the method moves to the last encoder layer and we report the resistance effect. Fourth, a dose-response curve over subspace size must show gains that level off rather than growing indefinitely. Endless growth would mean the method is simply rediscovering uniform shifting. If the second check fails on every task, we will state that causal selection adds nothing beyond feature-statistics alignment for geographic shift, which is itself a useful negative result for the steering literature.