Audited Early Exits for Satellite Imagery Under Geographic Shift

An early-exit network has small classifiers attached partway through its layers. When one of them is confident enough, the network stops and returns that answer, so easy inputs cost less compute. Confidence is usually measured with the entropy of the predicted class distribution: low entropy means probability is piled on one class. The stopping thresholds are tuned once, on data from the training regions. On satellite imagery deployed in a new continent, that tuning quietly breaks. Shallow layers respond to local texture, soil colour and roof materials, all of which change with geography, so early classifiers can become confidently wrong in one region and uselessly unsure in another. Nobody measures this, because the deployment region has no labels. This proposal makes the failure measurable without labels and then fixes it. In each small geographic cell, a few tiles are sent through the full network anyway. Comparing their early answers with the full-network answer gives a direct, label-free estimate of how often exiting early changes the result. Each cell's thresholds are then set so that this disagreement rate stays below a chosen bound with high probability. Because neighbouring tiles share terrain, sensor conditions and land use, audits from nearby cells can be pooled. This keeps the audit budget small.

The stakes are practical. Wall-to-wall mapping over petabytes of imagery is limited by inference cost, as argued in Position: Mission Critical – Satellite Data is a Distinct Modality in Machine Learning. The same paper reports accuracy drops of up to 30% at unseen sites. An operator who turns on early exiting to save compute currently has no way to know, region by region, how much accuracy the savings cost. The claim here is that a cheap spatial audit can give that number and enforce a bound on it, and that geographic autocorrelation is what makes the audit cheap.

Core Claim

The quantity that matters for an early exit is whether stopping changes the network's answer. Under geographic shift this can be estimated without labels, using the network's own final layer as the reference. For a spatial cell $g$, exit layer $l$ and entropy threshold $\tau$, define the disagreement rate

$$D_g(l,\tau) = \Pr\big(\hat y_l(x) \neq \hat y_L(x) \,\big|\, H_l(x) < \tau,\ x \in g\big),$$

where $\hat y_l$ is the prediction of the exit at layer $l$, $\hat y_L$ is the final-layer prediction, and $H_l$ is the entropy at layer $l$. A fraction $a$ of tiles in each cell is processed at full depth, which yields unbiased samples of $D_g$. We then pick, per cell and per layer, the largest $\tau$ whose upper confidence bound satisfies $\bar D_g(l,\tau) \le \varepsilon$. The bound is a one-sided binomial bound, chosen because each audited tile is a Bernoulli trial of agreement.

With a small audit budget a single cell has few audited tiles. We therefore pool counts from neighbouring cells with a spatial kernel. Its bandwidth is chosen by held-out audit likelihood, so that it shrinks toward per-cell estimates where the rate changes sharply and widens where it is smooth.

The empirical bet has two parts. First, the right threshold varies across continents far more than across nearby cells. Second, at equal total compute, including the audit, kernel-pooled audited thresholds will reach a given worst-region disagreement bound at lower cost than either one global threshold or independent per-cell thresholds.

This targets agreement with the full model, not label accuracy. That choice is deliberate. Early exiting is a compression of the full model, and its failure mode is unannounced divergence from it. Label accuracy is reported separately, so that cases where early exits help, the "overthinking" effect, remain visible.

Relation to Prior Work

The closest work is cross-domain early exiting in text.

CeeBERT tunes a single confidence threshold online on an unlabeled target domain. It treats the choice as a multi-armed bandit, rewarding confidence gained between the first exit and the exit layer minus a latency cost, and reports 2× to 3.1× speedups on NLP transfer tasks. Its reward is the confidence the model reports, which is exactly what spurious regional textures can inflate. It also assumes one stationary target domain with one global threshold. We replace the confidence reward with measured disagreement and replace the single threshold with a spatial threshold map.

DAdEE aligns every intermediate layer across domains with adversarial discriminators plus distillation, but keeps source-tuned thresholds, which its authors flag as suboptimal. That leaves room for our contribution. Its adversarial approach is also constrained by Extending the WILDS Benchmark for Unsupervised Adaptation, which found domain-invariance methods rarely beat plain empirical risk minimisation on real shifts, though FMoW was one of only two image tasks where adaptation methods clearly helped. We therefore treat representation alignment as an optional add-on, not the core.

Rethinking Calibration for Early-Exit Neural Networks shows that calibrating each exit in isolation is blind to the network's sequential structure, and that perfect per-exit calibration would make all exits identical. It proposes EEFP, a tiny learned predictor of whether continuing will change the outcome, evaluated on DomainNet. This motivates our target quantity, since EEFP also aims at "will continuing change the answer". It is also our strongest learned competitor. EEFP is trained once on source validation data and never re-checked in deployment. Our method could wrap EEFP's score instead of entropy, and we test that.

Two papers motivate the use of entropy without being competitors. Stopping Criterion Design for Recursive Bayesian Classification shows entropy can fall prematurely when trailing classes are pruned while the top two stay tied. This is relevant to fMoW's 63 classes, so we also audit a top-two margin score. PCEE-BERT adds a "patience" requirement, meaning several consecutive exits must agree, and is a natural score to plug in.

On the geographic side, spatial structure has already been exploited, but not for compute. STAR detects spatial clusters of errors with scan statistics and branches the network by coordinates. It needs labels and gives every tile in a region the same depth. We use spatial structure to set per-tile halting rules without labels. MMEarth-Bench adapts encoder weights at test time through multimodal reconstruction with geographic batching. It improves accuracy on a held-out Africa split but adds backward passes, which runs opposite to our goal of reducing cost. Trusting Small Training Dataset for Supervised Change Detection argues that diversity beats proximity for training data. That cuts against pooling neighbours, so the kernel bandwidth is learned, not assumed.

Research Program

The work has three stages.

The first stage is a measurement study. We train early-exit Vision Transformers and ResNets with exits at every block on FMoW-WILDS training regions, using the setup from WILDS built on Functional Map of the World. For every test tile and threshold we record the exit chosen, the early-versus-final disagreement, and correctness. We then map these by coordinates, compute variograms of per-cell optimal thresholds, and measure how much of their variance lies between continents, between cells within a continent, and within cells. This tells us whether a spatial method is warranted at all. If the variation is purely continental, a per-region threshold is enough and the kernel adds nothing. We also quantify the two predicted pathologies separately. Over-exit means confident shallow answers that the final layer overturns, concentrated in some regions. Under-exit means "futile" regions where nearly nothing exits early.

The second stage is the audited threshold controller described above. It has three parameters: audit fraction $a$, target $\varepsilon$, and confidence level. The controller can wrap any per-layer score, whether entropy, top-two margin, patience count, or the EEFP score. Audit cost is charged honestly, so audited tiles count as full-depth passes. The procedure needs one forward pass per tile and no gradients.

The third stage is a futility extension. In audited cells where early and final answers already agree on high-entropy tiles at some intermediate layer, those tiles can also exit there. This addresses the futile-sample trap: a futile tile that will be equally wrong at full depth costs less if it is answered at the layer where the final answer is already fixed.

Baselines

The naive reference is the source-tuned global entropy threshold, as in DeeBERT. It is tuned on the in-distribution validation set to a target speedup and applied unchanged everywhere. It shows what a practitioner gets today, and it is where per-region disagreement should blow up. The second reference is the full-depth network, the zero-savings anchor.

The strongest label-free online competitor is CeeBERT's UCB1 bandit, ported to images. It is run in two forms: one global bandit over the target stream, and one bandit per continent. It sees the same unlabeled tiles in the same spatial-sweep order as our method. Its latency-penalty parameter gets the same tuning budget on source validation that our $\varepsilon$ gets. The file contains no reported result on Lacuna for any early-exit method on FMoW-WILDS or other satellite data. The only figures available are CeeBERT's 2× to 3.1× speedups on text transfer, which are not comparable, so all comparisons will be run fresh.

The strongest learned competitor is EEFP from Rethinking Calibration for Early-Exit Neural Networks, trained on source validation trajectories and given the same backbone. That paper gives no per-domain numeric results to anchor against. For accuracy context only, a full-depth image DenseNet reached average F1 0.679 on fMoW, rising to 0.722 with metadata (Functional Map of the World).

The baseline that isolates our key ingredient is the same audit procedure with a single global threshold pooled over all target tiles, together with an independent per-cell version that does no pooling. We also run a per-region exit-rate matching baseline, which shifts each region's threshold so its exit fraction matches the source. This version uses no full-depth audits.

Beating a baseline means one of two things. At matched total FLOPs, including audit cost, our method has lower worst-region disagreement with the full model. At a matched worst-region disagreement bound such as 2%, it uses fewer FLOPs. In addition, its realised per-cell disagreement must exceed $\varepsilon$ no more often than the stated confidence level allows.

Why not just…?

Why not just run CeeBERT's bandit per region? This is a real threat on the efficiency axis, and per-region CeeBERT is a baseline for that reason. Its reward is the gain in the model's own confidence, so a region where shallow layers are confidently wrong will look like a good place to exit early. Nothing in the reward detects disagreement with deeper layers. The measurement study will show whether such regions actually exist in FMoW. If they do not, CeeBERT-per-region may match us, and we will report that.

Why not just use EEFP, which already predicts whether continuing helps? EEFP is the closest in spirit. Its predictor is fitted on source trajectories, and trajectories are exactly what geographic shift changes, so it inherits the same transfer problem as a fixed threshold, one level up. We test both EEFP alone and EEFP wrapped in our audit. If EEFP alone already holds the disagreement bound in every held-out region, the audit is unnecessary, and that is a clean negative result.

Why not just retune one global threshold on audited target tiles, or use a small static model? The global audit is our ablation. It wins if threshold variation turns out to be non-spatial, and the variogram analysis decides that before any method claim is made. A smaller static backbone gives up adaptivity on hard tiles. Following the input-blindness argument in PIVOT, we include a depth-truncated static model at matched FLOPs as a point on the cost–accuracy plot.

Proposed Experiments

The main benchmark is FMoW-WILDS with its region and year splits, using tile coordinates to define cells at several resolutions. We report six metrics: per-region and worst-region disagreement with the full model, average FLOPs including audit cost, worst-region label accuracy, the fraction of cells violating $\varepsilon$, and the cost–disagreement trade-off curve swept over $a$ and $\varepsilon$.

A secondary geographic test uses iWildCam2020-WILDS, grouping images by camera location. DomainNet serves as a non-spatial control. There, spatial pooling has nothing to exploit and should reduce to the global audit, which checks that any gains come from geography and not from the auditing itself.

The decisive checks are three. First, does a source-tuned threshold produce region-dependent disagreement on FMoW at all? Second, does kernel pooling beat both the global and independent per-cell audits at audit fractions of 1–5%? Third, does the stated bound hold on the held-out continents?

A fourth analysis compares entropy with top-two margin under auditing, to test whether the simplex pathology from the Bayesian stopping paper matters at 63 classes. If pooling gives no gain over the global audit, the paper's contribution reduces to the measurement study plus a label-free disagreement controller, and the spatial claim is dropped.