When a satellite-image segmentation model is moved to a region it was never trained on, such as a flood in a new country or a city with different roofs, accuracy can drop by up to 30%. Many of the pixels it now sees have feature values outside anything in the training data. A standard linear or neural classification head responds by extrapolating, and a feature value twice as large as anything seen in training produces a logit twice as confident. A decision-tree ensemble does not do this. Its output is a sum of leaf values learned from training data, so however strange a new pixel looks, the prediction stays within the range the trees already know. We propose to test whether this bounded extrapolation is the missing ingredient for label-free geographic transfer in dense mapping.
The system is simple. A frozen, pre-trained encoder supplies features at a resolution fine enough to resolve thin water channels and building edges. These are combined with raw physical measurements (radar backscatter, spectral reflectance, water indices), and a gradient-boosted tree ensemble, meaning many small decision trees each trained to correct the errors of the previous ones, classifies every pixel. Nothing uses target-region labels or target imagery, and all tuning is done by holding out whole source regions. The claim is falsifiable and specific. If the tree head beats a linear or small neural head fed the identical inputs, and the advantage is concentrated on pixels whose features fall outside the source range, then the choice of classifier matters for geographic robustness. If not, it does not.
Core Claim
Earlier hybrid "deep features plus tree classifier" systems were motivated by small data. Object Recognition from very few Training Examples for Enhancing Bicycle Maps trained a Random Forest on CNN features with about 15 examples per class. Small Defect Detection Using Convolutional Neural Network Features and Random Forests fed intermediate U-Net features to a Random Forest and beat the end-to-end softmax U-Net on weld defects (84.5% detection at 2.4 false positives per image). Both evaluated only in-distribution, and neither offers a reason why trees should help under shift. We supply that reason. Let $F_{\text{tree}}(x)=\sum_{m=1}^{M} v_{m,\ell_m(x)}$ be the ensemble output, where $v_{m,\ell}$ is the value of leaf $\ell$ in tree $m$ and $\ell_m(x)$ is the leaf that input $x$ lands in. Then for every input,
$$|F_{\text{tree}}(x)| \le \sum_{m=1}^{M} \max_{\ell} |v_{m,\ell}|,$$
whereas a linear logit $w^\top x$ grows without limit as $x$ moves off the training support. Under geographic shift, where Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning documents drops of up to 30%, we hypothesize that a large share of errors are confident extrapolations on off-support pixels. Trees avoid these because they fall back on axis-aligned splits over the few physically stable inputs, for example a backscatter threshold for open water.
This hypothesis explains why the three other ingredients have to be present. High resolution matters because the off-support pixels that matter most are thin structures, and these are smeared away at the 1/16 token resolution of a ViT-style encoder. The absence of target labels matters because a tree head cannot be adapted by entropy minimization at test time, so robustness has to be built in before deployment. Raw physical covariates matter because trees can split on unnormalized quantities with physical units. This is exactly the signal that instance normalization would erase.
Research Program
The per-pixel feature vector has three parts. The first is frozen foundation-model tokens, bilinearly upsampled to full resolution. The second is a full-resolution feature stack from a small network built from densely stacked dilated convolutions, following Dense Dilated Convolutions Merging Network for Semantic Mapping of Remote Sensing Images. DDCM-Net keeps features at 1/8 resolution with about 10M parameters. We train our small variant on source data only and then freeze it. The third part is raw bands and simple indices. We train gradient-boosted trees on a class-balanced subsample of source pixels, which keeps memory bounded. For inference on large tiles, we compile the trained ensemble into convolution-like operations, following the tree-to-network mapping of Reinders et al. That mapping cut their inference from over 10 hours to 6.08 seconds per image.
All hyperparameters are chosen by leave-one-region-out validation over source regions only. These include tree depth, number of trees, learning rate, feature subsampling, which encoder layer to use, and the dilation rates. Sagawa et al. warn in Extending the WILDS Benchmark for Unsupervised Adaptation that many adaptation methods quietly rely on labeled target validation data to pick hyperparameters. Our protocol forbids this, and we apply the same rule to every baseline.
The analysis component measures off-support-ness for every target pixel. We use two measures: the fraction of its features outside the source 1st–99th percentile range, and its Mahalanobis distance to source features of the predicted class. We then plot error against this score for each head. The core claim predicts that the tree and linear heads perform about equally on in-support pixels and that the gap opens on off-support pixels.
We also address the metadata tension in the literature. In Functional Map of the World, metadata raised F1 from 0.679 to 0.722. In Fine-tune Smarter, Not Harder, coordinates and timestamps added negligible gain to foundation models. We add latitude, longitude and acquisition date as tree inputs, and predict they help in-distribution but not on held-out regions. Coordinates of an unseen region are off-support by construction, so bounded trees should neutralize them while linear heads may be harmed by them.
Baselines
The naive reference point is the frozen foundation model, Prithvi 2.0 300M, with a linear per-pixel head trained on source data. It uses the same frozen encoder, the same source pixels and the same leave-one-region-out tuning budget. It shows what the frozen features give with no special head, no high-resolution branch and no covariates.
The strongest existing method is LoRA fine-tuning of Prithvi 2.0, as evaluated in Fine-tune Smarter, Not Harder. LoRA inserts small trainable low-rank matrices into a frozen model. That paper reports 87.57% mIoU on the Sen1Floods11 Geographic Hold-Out Set against 82.07% for full fine-tuning. It is the strongest competitor because it already shows that keeping pre-trained features mostly frozen is what protects geographic generalization, and our proposal also keeps the encoder frozen. We will rerun LoRA ourselves with the same training split, number of seeds and GPU-hour budget, and tune it by leave-one-region-out rather than any target-derived choice. We report both our rerun and the published 87.57%. Beating it means exceeding our rerun's mean hold-out mIoU by more than the across-seed spread, with zero target labels. Matching it while training only a tree head on frozen features would still be a useful but weaker result, and we will say so.
We also include full fine-tuning (82.07% reported on the same hold-out set) and a test-time adaptation baseline. The latter is Efficient Test-Time Model Adaptation without Forgetting (EATA) applied to the LoRA model on streaming hold-out scenes. EATA updates the model by entropy minimization on confident, non-redundant test samples with a penalty against forgetting. It reduced ImageNet-C severity-5 error to 65.0% from Tent's 71.6%. EATA uses unlabeled target imagery that our method never sees, so a win over it is a strong result.
For the second benchmark, cross-city ISPRS transfer, we found no reported cross-city result on Lacuna for DDCM-Net or any deep segmenter under our protocol. The closest reference is the cross-dataset collapse of HF-FCN from 0.74 to 0.10 building F-score reported in Accurate Building Detection in VHR Remote Sensing Images using Geometric Saliency. We therefore train an end-to-end DDCM-Net ourselves as the reference there. We also treat the unsupervised Geometric Building Index (0.59 F-score on Potsdam) as a no-learning floor for buildings.
The ablations isolate the key ingredient. We swap the tree head for a linear head and for a two-layer neural head on identical inputs. We drop the physical covariates. We drop the full-resolution branch, leaving upsampled 1/16 tokens only. We also include a clipped-feature linear head, in which every input is clamped to the source percentile range before a linear classifier. That last variant tests whether bounding inputs is enough, or whether trees add something beyond clamping.
Why not just…?
Why not just put a neural or linear head on the same features and covariates? This is the real threat. If a two-layer head with the same concatenated inputs matches the tree head, the gain comes from the covariates and resolution, not from trees. The ablation is designed to show exactly that. The distinguishing prediction is the off-support error curve: the tree advantage should grow with off-support-ness and vanish on in-support pixels. The clipped-feature linear head further separates "bounded inputs" from "bounded, axis-aligned outputs." If clipping recovers most of the gain, we will report that the lesson is output boundedness, which is cheaper than trees.
Why not just use LoRA, which already scores 87.57% on the geographic hold-out? LoRA is the bar we must clear, and we may fail to clear it. It still ends in a linear head at token resolution that extrapolates on off-support pixels, and it cannot use unnormalized physical covariates without retraining the head. We also run a combination, trees on LoRA-adapted features, so that a complementary gain is visible if one exists.
Why not just adapt at test time with EATA? Test-time adaptation needs a stream of target imagery and can drift under sustained shift. It also inherits the hyperparameter problem Sagawa et al. describe, since confidence thresholds are hard to set without target labels. In WILDS, many unsupervised adaptation methods failed to beat plain training on the source data, though Camelyon17 improved from 70.8% to 86.7% with Noisy Student, so gains are dataset-dependent. We run EATA directly as a baseline rather than assuming it fails.
Why not just add instance normalization, as DualNorm does? Frustratingly Easy Person Re-Identification raised zero-shot PRID Rank-1 from 27.2% to 60.4% by stripping low-level style with instance normalization. In satellite imagery, however, absolute reflectance and backscatter carry physical meaning, such as whether a surface is water. We include an instance-normalized encoder variant as a baseline. We predict it hurts water segmentation specifically, and we will check this against per-class IoU.
Decisive Experiments
The primary benchmark is the Sen1Floods11 Geographic Hold-Out Set, using Sentinel-1 VV/VH, Sentinel-2 bands and NDWI/MNDWI as covariates, and reporting mIoU and water IoU. The secondary benchmark is ISPRS Potsdam→Vaihingen and the reverse, using the shared IRRG bands and reporting per-class F1 with emphasis on car and building boundaries. The primary ablation, tree versus neural head on identical inputs, is run across five seeds.
Three results decide the claim. The first is whether the tree head beats the neural and linear heads on held-out regions while roughly tying them in-distribution. The second is whether that gap is concentrated in the highest off-support bins of the error curve. The third is whether the full-resolution branch specifically improves thin-structure classes on held-out regions. If the first two fail, the bounded-extrapolation hypothesis is wrong, and we will report the head-choice question as settled negatively for geographic shift.
References
- Object Recognition from very few Training Examples for Enhancing Bicycle Maps
- Small Defect Detection Using Convolutional Neural Network Features and Random Forests
- Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning
- Dense Dilated Convolutions Merging Network for Semantic Mapping of Remote Sensing Images
- Extending the WILDS Benchmark for Unsupervised Adaptation
- Functional Map of the World
- Fine-Tune Smarter, Not Harder: Parameter-Efficient Fine-Tuning for Geospatial Foundation Models
- Efficient Test-Time Model Adaptation without Forgetting
- Accurate Building Detection in VHR Remote Sensing Images Using Geometric Saliency
- Frustratingly Easy Person Re-Identification: Generalizing Person Re-ID in Practice