Footprint-and-Context Pretraining for Auditable Building Damage Assessment

Satellite damage-assessment models look at a building before and after a disaster and output one of four grades, from undamaged to destroyed. They give no account of why. The question a responder most needs answered is simple: was a building called "destroyed" because its roof is gone, or because it sits next to floodwater? A single score cannot tell a real structural finding from an inference drawn from the neighborhood. We propose a model that answers this by construction. A self-supervised encoder is pretrained to reconstruct hidden pieces of the building footprint and, separately, of a thin ring of surroundings. The damage prediction is then written as the sum of a "building evidence" term and a "context evidence" term, and each term can be read and tested on its own.

The bet is that a spatial split of the evidence is cheap in remote sensing and useful there in a way it is not in medicine. It is cheap because building outlines already exist at scale: xBD ships footprint polygons, and OpenStreetMap footprints cover much of the world. It is useful because in disasters the context outside the ROI (the region of interest, here the building footprint) is sometimes real evidence, such as fire, water, or debris, and sometimes a shortcut. Existing methods either throw that context away or blend it in invisibly.

Core Claim

We make three connected claims. First, guiding masked pretraining with building footprints and a context ring gives an encoder whose features, pooled separately inside and outside the footprint, support a damage classifier with an additive inside/context structure. This classifier should match an unconstrained fine-tuned head in accuracy, especially when labels are scarce. Second, the decomposition is faithful: if we swap a building's surroundings for those of a building with a different damage grade, the change in prediction flows almost entirely through the context term. Post-hoc attribution on a black-box model does not reliably give this property. Third, the per-disaster share of evidence coming from context predicts where the model will fail when it is moved to an unseen disaster. That makes the decomposition a diagnostic tool and not just a picture.

The novelty is not "region-guided masking for satellites" alone. That would be a direct port of RGMIM, and RGMIM's own gain over plain masked pretraining was small. The novelty is in making the pretraining region structure and the classifier's evidence structure the same object, so that the explanation is a measurable, falsifiable part of the model.

Relation to Prior Work

Two works supply the pieces we reuse. RGMIM masks 75% of patches inside a lung mask and reconstructs only those pixels, improving chest X-ray accuracy over standard masked autoencoding (0.962 vs 0.956; 0.771 vs 0.754 at 1% labels). It fine-tunes an opaque classifier, needs expert masks at pretraining time, and discards everything outside the organ. We borrow its masking mechanism and change three things. We add an explicit context ring. We test predicted and coarse masks in place of expert ones. We carry the region structure into the classifier. Lesion-Centric Latent Phenotypes supplies the mask-weighted pooling idea, which averages encoder features only inside a predicted lesion mask. It used a small supervised segmentation encoder and pooled only inside the ROI. We pool in two disjoint regions on top of a self-supervised encoder.

Two works motivate the problem. Self-Supervised Learning for Building Damage Assessment shows that in-domain self-supervision (DINO contrastive learning plus local reconstruction) beats ImageNet on xBD at 1% labels. Its pretraining ignores footprints, and its damage head is a black box. DisasterInsight reorganizes xBD into about 112,507 building-centered instances and fine-tunes vision-language models to write reports. Its explanations are generated text, not a structural property of the classifier.

Two findings constrain the design. The first is the "distraction effect" in Integrating Clinically-Relevant Features into Skin Lesion Classification: forcing predictions through hand-defined clinical concepts hurt accuracy on benign lesions. This steers us away from a concept bottleneck built on annotated attributes, such as roof-collapse percentage, toward a spatial decomposition that needs no concept labels. The second is the xBD annotation effort (xBD, Creating xBD), which explicitly recorded water, fire, smoke, and lava as damage-relevant context. This is why hard masking in the style of mask-weighted GAP for CT triage or pixel masking in echocardiography is the wrong default here, even though those methods do suppress the shortcut learning documented in lung ultrasound.

Research Program

Pretraining. The encoder is a Vision Transformer (a network that splits an image into patches and lets every patch attend to every other). It takes the pre- and post-disaster tiles together. Each footprint polygon defines two regions: the footprint itself, and a context ring made by dilating the footprint by a fixed ground distance and subtracting it. The ring distance is swept from roughly 5 to 30 meters, about 10 to 60 pixels at xBD's ~0.5 m resolution. On the post-disaster tile we mask a high fraction of footprint patches, a lower fraction of ring patches, and a small fraction of the remaining background. The pre-disaster tile stays largely visible. Reconstruction loss is computed separately in each region, with weights $\lambda_{in}, \lambda_{ctx}, \lambda_{bg}$ set so building pixels are not drowned out by background. Because the pre-disaster view is visible, the network must learn what changed at a structure and in its surroundings rather than learn to paint generic terrain.

Footprint sources. We pretrain with three kinds of footprints and compare them directly. The first is ground-truth xBD polygons. The second is footprints predicted by a segmenter trained on a small labeled subset. The third is coarse versions: dilated or bounding-box footprints. This directly tests whether the region guidance survives noisy masks, which RGMIM left open.

Classifier. For building $b$, we pool the encoder's features inside the footprint to get $z_{in}$ and inside the ring to get $z_{ctx}$. The damage logits are

$$s_k(b) = w_k^{\top} z_{in} + u_k^{\top} z_{ctx} + c_k,$$

where $k$ indexes the four damage grades, $w_k$ and $u_k$ are per-grade weight vectors, and $c_k$ is a bias. Because the two terms simply add, every prediction reports how much of its margin came from the building and how much from the context.

We also train two regularized variants. One adds a penalty on the context term's share of the margin. The other adds shallow nonlinear heads per region while keeping the sum additive. Together they let us trace the accuracy-versus-transparency curve instead of fixing a single point on it.

Baselines

Naive reference points. The weakest reference is the original xBD baseline, reported at weighted F1 0.2654. Alongside it we run an ImageNet-initialized encoder of the same architecture with a standard unconstrained head. For this second baseline, Self-Supervised Learning for Building Damage Assessment reports F1 0.321 at 1% labels.

Strongest existing method. This is the asymmetric DINO-plus-reconstruction encoder from Self-Supervised Learning for Building Damage Assessment, with its Siamese UPerNet head. It reports F1 0.390 at 1% labels and 0.636 at 20% labels, against 0.366 for standard DINO. It is the right competitor because it is in-domain, self-supervised, bitemporal, and evaluated on the same benchmark in the label-scarce regime where pretraining matters most.

We will re-run it under our protocol rather than only quote it. That means the same unlabeled xBD pool, the same encoder size, the same pretraining compute budget, the same labeled splits at 1, 5, 20, and 100%, and the same number of tuning trials per method. We report both our reproduced number and the published one.

For per-class comparison we use the weakest-class figure from PPM-SSNet: minor-damage F1 of 0.41. For the building-centered DisasterInsight tasks, the paper reports DI-Chat as having the lowest ordinal error, but we found no reported numeric result on Lacuna for DI-Chat damage MAE, so we will evaluate our model on DisasterInsight's damage-level task without a quoted target.

What counts as beating the strongest method. We need two things. First, overall F1 at 1% and 5% labels that matches or exceeds the reproduced DINO-plus-reconstruction number, with confidence intervals from at least three seeds. Second, a context-swap faithfulness score that the baseline plus post-hoc attribution does not reach.

Isolating ablations. Three ablations test the key ingredients.

The first isolates the pretraining. We attach the same additive head to encoders pretrained with plain uniform masking, footprint-only guided masking in the RGMIM style, and footprint-plus-ring masking. If the additive head works equally well on any encoder, the pretraining contribution is null and we will say so.

The second isolates the classifier structure. We compare the additive head against a single pooled vector over footprint plus ring, which keeps the same region access but loses the decomposition.

The third isolates each region's value. We run footprint-only and ring-only classifiers to measure how much each region carries on its own.

Why not just…?

Why not just take the existing SSL encoder and add mask-weighted pooling at fine-tuning time? This is the most serious threat. The additive head needs footprints only at fine-tuning, so region-aware pretraining could turn out to be unnecessary. RGMIM's small margin over plain masked autoencoding (0.962 vs 0.956) makes this plausible. The first ablation above settles it directly. If unguided encoders give equal accuracy and equal faithfulness, the contribution narrows to the decomposed head and its diagnostic use, and we will report it that way. We expect guided pretraining to matter mainly at 1% labels and for the context term's quality, because an unguided objective spends most of its reconstruction effort on roads, vegetation, and soil.

Why not just run a black-box model and split a Grad-CAM or occlusion map into inside and outside the footprint? Post-hoc maps summarize gradients or occlusion effects. They are not a guarantee about how the prediction was formed, and occluding a ring pushes the input off the distribution the model was trained on. The context-swap test gives a direct comparison. We replace a building's ring with the ring of a building of a different grade, then compare how much of the resulting prediction change each method attributes to context. For our model the answer is read off $u_k^{\top} z_{ctx}$, though leakage through self-attention can still move $z_{in}$, and we measure that leakage. For the black box, the answer comes from the attribution map. If post-hoc splitting tracks the true swap effect just as well, the case for the built-in decomposition weakens to convenience, and we will report that.

Why not just crop each building with some padding, as DisasterInsight does, and let the model use context freely? Padded crops do keep the context. But they give no account of how much the decision relied on it, which is precisely the failure we target. We include a padded-crop classifier with the same encoder as a baseline. If it wins on accuracy, the question becomes whether the gap is worth the lost auditability. That is exactly the tradeoff the regularized variants trace out.

Why not a concept bottleneck with damage attributes? xBD has no dense attribute labels. The skin lesion study shows concept constraints can cost accuracy on the common classes, which here would be the undamaged majority. The spatial split needs no new labels.

Proposed Experiments and Decisive Checks

All experiments use xBD's bitemporal tiles. We report per-class F1 for all four grades and the combined F1 used by the Self-Supervised Learning for Building Damage Assessment paper, at 1, 5, 20, and 100% labels, over three seeds. We also report ordinal error on DisasterInsight's building-centered damage task.

Context-swap faithfulness. On held-out buildings, we swap in surrounding rings from other buildings with matched ground-truth grades and with mismatched ones. We measure the fraction of the prediction change carried by the context term, and how far the inside term moves under the swap.

Leave-one-disaster-out generalization. We hold out each of xBD's 19 events in turn. This is the decisive diagnostic check. For each held-out disaster, we correlate the context term's share of the margin, measured on the training disasters of the same type, with the accuracy drop on the held-out event. If a high context share predicts transfer failure, the decomposition does real diagnostic work. If it does not, it is just a picture.

Mask robustness. We repeat the main 1%-label comparison with ground-truth, predicted, and coarse footprints in pretraining. This tells us whether the method holds up when masks are not expert-quality.

What would falsify the proposal. Any one of the following does it. Guided pretraining gives no gain over unguided pretraining under the same additive head. Post-hoc splitting of a black box matches the built-in faithfulness. The context share has no relationship to cross-disaster failure.