Spending a Verification Budget Well When a School Detector Moves to a New Country

Satellite-based mapping can now propose thousands of candidate schools, power poles, or other facilities that are missing from official registries. Each candidate still has to be confirmed by a human analyst or a field visit before anyone acts on it, for example by connecting a school to the internet. In one national deployment of a weakly supervised school detector, 5,369 of 12,352 predictions in Senegal had no match in the government registry and were left for on-the-ground validation (Large-Scale School Mapping). Verification is the real bottleneck. The usual policy is to verify the highest-scoring candidates first, but that policy fails exactly when the detector has been moved to a country it was not trained on. Its scores then become wrong in uneven ways: reliable in some landscapes and badly overconfident in others.

We propose to treat verification in a new country as a sequential decision problem, specifically a contextual bandit, meaning an agent that repeatedly picks one option, observes whether it paid off, and updates its beliefs. Here the agent picks which candidate to verify next. Its context is a description of the candidate's surroundings as a mixture of landscape types, found without labels by applying multi-topic decomposition to image embeddings. The claim is that one verification budget can do two jobs at once. It can find more true unregistered facilities per check than score-ranked verification, and because the agent keeps a known amount of randomness in its choices, it can also give an unbiased, landscape-by-landscape estimate of how accurate the detector really is in the new country. That second output addresses a problem the cross-dataset validation literature leaves open: how to measure performance on a target distribution you have no validation data for.

Core Claim

Under cross-country shift, a detector's precision depends on landscape type: dense informal settlement, peri-urban mix, rural agriculture, and so on. Most tiles are mixtures of these types rather than members of one. A bandit whose context is the tile's mixture weights over unsupervised landscape topics can learn the target country's precision surface from a few hundred verifications. Ranking candidates by the detector score adjusted with that learned correction should yield more confirmed facilities per verification than ranking by raw score. Logging the probability with which each candidate was chosen then lets us reweight the verified outcomes into an estimate of target-country precision for each landscape, with valid confidence intervals.

Where This Sits in Prior Work

The infrastructure papers motivate the problem. The school-mapping pipeline classifies 300 m tiles with a transformer ensemble and localizes schools with activation maps. Its reported in-country accuracy is very high (AUPRC up to 0.998), but each country gets its own model and the leftover candidates are verified passively (Large-Scale School Mapping; a related human-in-the-loop deployment in Colombia and the Caribbean is Yi et al.). We do not change the detector. We change how its output is checked. PGRID shows the shift we care about at a smaller scale. Pole detection reaches F1 0.66 on US and New Zealand satellite imagery and 0.71 in Kenyan refugee camps, and the paper reports fences and clotheslines as systematic false positives. Errors of that kind depend on landscape, which is what our context is meant to capture.

The cross-dataset validation papers constrain the design. WILDS shows that satellite models lose substantial accuracy across countries (PovertyMap Pearson correlation drops from 0.60 to 0.45). It also shows that invariance penalties rarely beat plain training, and that you cannot tune for a target domain you cannot see. Compositional Feature Alignment separates class and domain directions with orthogonal heads, but it needs discrete domain labels. We take a different route. We accept that the detector will be imperfect under shift and spend target-domain labels where they best correct it and measure it. We also need no domain labels, because the landscape topics are discovered.

The topic-modeling papers supply the context representation. Semantic Component Analysis clusters text embeddings, subtracts each cluster's mean direction, and clusters the remainder again. This gives each document several topics and leaves almost no documents unassigned. We apply the same peeling to frozen image embeddings of tiles, so a tile gets loadings on, say, "dense roofs", "irrigated fields", and "bare ground" together. That is the mixed-use structure that the single-label k-means in urban phenotyping cannot express. We do not use the optimal-transport regularizer of ECRTM, because its equal-size topic assumption suits landscape types poorly: rare types such as water would be over-expanded.

The sequential decision papers supply the update rule and one key contrast. Verification outcomes are noisy because registries are spatially offset from imagery, so a "no match" sometimes means a real school recorded a few hundred meters away. We therefore fit the correction model with the one-pass Huber update of Hvt-UCB. It limits the influence of such mislabeled outcomes at constant time and memory per step, which matters when every one of up to millions of national candidates is re-scored after each verification. The bounded binary reward means the full heavy-tail machinery of Catoni bandits is not needed in the main setting. It becomes relevant only in a variant where the reward is the number of facilities confirmed in a tile. The contrast is with DT-ECE, which treats a new task as one member of a known, well-separated library and identifies it before exploiting. The closest analogue here would be "find which source country's calibration the target resembles." Our mixture context is designed for the open-world case where the target country resembles no single source country.

Research Program

The pipeline has four parts. First, a detector is trained on source countries; we reuse the published architecture family. It is then applied to a held-out target country to produce candidates with scores $s$. Second, every tile gets a landscape mixture vector $z \in \mathbb{R}^K$. These come from iterative component peeling on frozen DINOv2 or CLIP embeddings fitted on unlabeled imagery from all countries, with $K$ chosen by held-out reconstruction error and topic diversity. Third, the bandit models the probability that a candidate is a true facility as a logistic function of the detector's logit plus a learned linear correction, $\mathrm{logit}(s) + \theta^\top [z, z \cdot \mathrm{logit}(s)]$. The parameters $\theta$ are updated by one-pass Huber descent after each verification. At each step the agent verifies the candidate with the highest optimistic estimate, except that with probability $\epsilon$ it draws a candidate from a landscape-stratified uniform distribution. This keeps every candidate's selection probability known and bounded away from zero. Fourth, the logged probabilities feed an inverse-propensity-weighted estimator of target precision, overall and per dominant landscape topic.

We will also analyze where the gain comes from. If the target-country precision surface is flat in $z$, the correction is pointless and greedy ranking by score is optimal. Measuring how much precision actually varies with landscape in each held-out country is therefore a finding in its own right, and it predicts when the method should help.

Baselines

The naive reference is uniform random verification. It is the worst way to find facilities and the textbook way to estimate precision without bias, so it anchors both of our metrics.

The deployment default is greedy verification by raw detector score, which is what the published pipeline implies for its unmatched candidates. No reported result was found on Lacuna for verification yield under cross-country transfer for this or any other method. The only reported numbers are in-country: AUPRC above 0.96 per model and up to 0.998 for the ensemble (Large-Scale School Mapping). These describe the source-domain ceiling that greedy ranking enjoys before transfer.

The strongest competitor is calibrate-then-exploit. It spends the first $m$ verifications on a landscape-stratified random sample, fits the same logistic correction once, then ranks greedily for the rest of the budget. This shares our context, our model, and our estimator; the only difference is that its allocation is not adaptive. To keep the comparison fair, $m$ is tuned on source-country simulations with the same tuning budget we give $\epsilon$ and the Huber threshold. Beating it means a higher discovery curve at equal budget in a majority of held-out countries, with no worse precision-estimate coverage.

Two ablations isolate the key ingredient. The first is the same bandit with raw 768-dimensional embeddings projected to $K$ dimensions by PCA instead of topic loadings. The second uses hard k-means phenotype indicators in the style of Metzler et al.. Together they ask whether multi-membership landscape structure, rather than just any low-dimensional context, is what helps.

A closed-world baseline in the spirit of DT-ECE keeps one fitted correction per source country, identifies the best-matching one from early verifications, and then exploits it.

Finally, ordinary least-squares updates in place of the Huber update test whether robustness to registry-offset label noise matters.

Why not just…?

Why not just verify the highest-scoring candidates first? In-country this is nearly optimal, given AUPRC near 0.998. The premise of the proposal is that it degrades unevenly after transfer, as the WILDS cross-country drops and PGRID's landscape-specific false positives suggest. If our per-landscape precision analysis finds a flat surface in a held-out country, greedy will tie us there, and we will report that rather than claim a gain.

Why not just recalibrate on a small random sample and then rank greedily? This is the serious threat. It may capture most of the benefit whenever the precision surface is simple and stable. Adaptivity should matter when some landscape types are rare among candidates, so that a fixed random batch barely samples them, or when the useful correction concentrates in a few topics. The calibrate-then-exploit baseline is built to settle this directly. If it matches the bandit, the contribution narrows to "topic-mixture calibration plus randomized logging gives free target-domain validation." That is still useful, but it is a smaller claim, and we would state it as such.

Why not just use uncertainty sampling to retrain the detector? Uncertainty sampling targets what the model is unsure about, not candidates likely to be real facilities. It therefore spends a discovery budget on ambiguous cloud or registration artifacts, and it gives no unbiased precision estimate. Retraining is complementary to our method, not a rival. We will include a greedy variant that fine-tunes the detector every 200 verifications, to test whether the cheap correction over the topic context is competitive with retraining the detector itself.

Why not just learn domain-invariant features, or use CFA with country labels, and skip verification design? Better transfer shrinks the problem but does not remove it. WILDS reports that invariance methods rarely beat plain training on real shifts, and any remaining error still needs measuring in the target country. As a stronger detector condition, we will run our allocation on top of a CFA-fine-tuned detector that uses country as the domain label.

Proposed Experiments

The primary testbed is leave-one-country-out replay on the ten-country school dataset, assuming access to the released tiles and registry matches (Large-Scale School Mapping). The registry, with a spatial tolerance, serves as the verification oracle. Senegal's 5,369 unmatched candidates are held aside as a qualitative check, since we cannot score them. The second testbed is the region split of FMoW-WILDS and Functional Map of the World, with one-vs-rest detection of an infrastructure category and ground-truth labels as the oracle. The third is the PGRID transfer from the satellite benchmark to Turkana drone imagery for pole candidates; it stresses the method under a sensor change as well as a geographic one.

We will report two metrics. The first is discovery yield, the number of confirmed facilities as a function of verifications, summarized by area under that curve at budgets of 1% and 5% of candidates. The second is precision-estimation quality: absolute error and 90% interval coverage of overall and per-topic precision estimates against the full oracle. Every method runs on identical candidate pools, detectors, and budgets, over 20 random seeds per held-out country.

Three checks are decisive. First, the bandit must beat calibrate-then-exploit on yield in most held-out countries while keeping interval coverage near nominal. Without that, the adaptive part of the proposal fails. Second, topic-mixture context must beat both PCA and hard-cluster context. Without that, the topic-modeling ingredient fails. Third, across countries, the bandit's gain over greedy ranking should track the measured variance of precision across landscape topics. That relationship would show the mechanism is the one we claim, rather than an artifact of tuning.