Erased from the Map, Recoverable in 3D: Height-Consistent Anonymization of Overhead Imagery

When a sensitive building, such as a private compound or an industrial site, is painted out of a satellite image, the edit usually happens in two dimensions: a mask over the roof footprint, filled by an image generator. But an overhead photo is a flattened projection of a 3D scene, and the flattening leaves traces of height behind. Shadows stretch according to building height and sun angle. Tilted ("off-nadir") views make tall structures lean and expose their walls. Neighboring images of the same place taken from other angles show the building again. We propose to test whether modern monocular height estimators and diffusion inpainters can use these traces to reconstruct an erased structure's footprint and height. We also propose a defense: instead of painting over a 2D mask, the sanitizer invents a plausible decoy structure with its own height, then renders that decoy's roof, lean, and shadow consistently in every view that is released.

The motivating analogy comes from medical imaging. Removing facial voxels from head MRIs does not prevent re-identification, because a diffusion model can regrow the face from the underlying skull. The regrown faces reach 0.38 mm mean surface distance, compared with 0.71 mm for a population-average face. Removal also destroys downstream research signal (Pitfalls of defacing whole-head MRI). That paper shows the "worst of both worlds" failure but offers no replacement method, and it never leaves neuroimaging. Our claim is that overhead imagery has the same structure. The 2D image is a projection in which height is entangled with shadow and lean, so 2D sanitization leaks exactly what projection hides imperfectly. The fix is to sanitize the 3D cause rather than its 2D appearance.

Core Claim

We make two falsifiable claims. The first is an attack claim: on multi-angle satellite imagery, an attacker who combines a remote-sensing height estimator with a learned restoration model can recover an erased building's footprint and height from 2D-inpainted images. The recovery will be substantially better than a neighborhood-average guess, and much better when an unsanitized off-nadir view of the same site is available. The second is a defense claim: replacing the building with an explicit decoy height map, rendered with view-correct lean and sun-correct shadows and shared across all views, drives recovery back to the neighborhood-average level. It does this while keeping downstream building segmentation on the rest of the scene closer to the unsanitized level than blurring or dilated-mask inpainting does.

If the attack claim fails, and 2D inpainting already leaves nothing recoverable, that negative result is itself useful. It would tell practitioners that simple pipelines suffice, and we would report it as such.

How This Differs from Prior Work

Generative anonymization so far operates on faces or on ground-level scenes. Examples include identity-swapping GANs (CIAGAN), identity-invariant representations for expression recognition (VGAN-based representation learning), and diffusion face swapping with a single reconstruction loss (Face Anonymization Made Simple). The closest system is SVIA, which segments street scenes, inpaints buildings and roads with a latent diffusion model, and harmonizes the result. It shows that background structure leaks city identity, with 99.2% re-identification accuracy on raw Cityscapes. SVIA motivates our work because it proves context leaks. It also supplies our strongest baseline pipeline. It works purely on 2D pixel masks from horizontal cameras, and it lists cross-view consistency as an open limitation. That limitation is precisely what we target.

On the projection side, UVTR shows in driving scenes that keeping explicit height (11 vertical bins instead of 1) improves camera detection by 4.2% NDS. That result supports our choice to represent the decoy as geometry rather than texture, though UVTR has nothing to do with privacy or aerial views. SpaceNet MVOI shows that building segmentation collapses off-nadir, with TernausNet F1 falling from 0.62 at nadir to 0.22 beyond 40°. Its downsampling controls show the drop is caused by geometry rather than resolution. That evidence both motivates the leak hypothesis and supplies our data. RS3DBench and D3-RSMDE supply the attacker's tools: diffusion-based monocular elevation estimators for overhead images. Both target terrain rather than discrete buildings, so we must fine-tune them. D3-RSMDE's finding that more than 6 denoising steps makes the model invent spurious texture is a constraint on the attacker. It also cuts in our favor: a hallucinating estimator is a weaker attacker, so we will tune the step count as part of the attacker's budget.

Research Program

Data and threat model. SpaceNet MVOI is the main testbed. It covers 665 km² of Atlanta at 27 look angles from −32.5° to 54° off-nadir, with 126,747 building footprints. For each experiment we designate a random subset of buildings as sensitive. The file lists no per-building height ground truth for MVOI, so we will derive reference heights from multi-view geometry: roof displacement relative to the ground footprint grows with look angle in proportion to height. We will validate these heights against RS3DBench-style elevation pairs where coverage overlaps. This derived reference is an explicit risk, and we will report its own error.

The attacker receives the released images, which may be sanitized nadir views, sanitized off-nadir views, or a mix. The attacker also knows the sanitization method, can train on sanitized and unsanitized pairs from other city tiles, and outputs a footprint mask and height for each sensitive site.

Attacker. The attack follows the refacing design of the MRI study, adapted to aerial geometry. A height estimator (D3-RSMDE backbone, fine-tuned on MVOI with derived heights) predicts elevation over the sanitized region and its surroundings. A conditional diffusion restorer, trained on paired tiles, then regenerates the original building conditioned on the sanitized image, the predicted height map, and the view and sun metadata. A cross-view variant additionally aligns other released angles of the same site.

Defense. The sanitizer first samples a decoy footprint and height from a prior fit to the surrounding block. It then renders, for each released view, three things: the roof region, the lean displacement implied by the decoy height and look angle, and the shadow polygon implied by the decoy height and sun angle. Finally, it inpaints with a latent diffusion model conditioned on these rendered geometry maps rather than on a bare mask, followed by a light harmonization step borrowed from SVIA. The same decoy is used in every view of the site.

The key ingredient is that the edited region and its conditioning are derived from a 3D decoy projected into each view, not from a 2D footprint. The privacy target moves from "the pixels look different" to "any geometry recovered from these pixels is the decoy's."

Baselines

The naive reference points are releasing the unaltered images and blacking out or Gaussian-blurring the oriented footprint region. The unaltered images give the attacker's ceiling and the utility ceiling. Blackout and blur represent current practice. We expect them to hurt downstream utility, just as defacing did for MRI. We will also test whether shadows outside the mask still reveal height.

The strongest existing method is SVIA's pipeline re-trained on overhead imagery: segmenter, latent diffusion inpainter with its noise injection, and harmonizer, all fit on MVOI training tiles with the same diffusion backbone, compute budget, and hyperparameter search budget as our defense. No reported result for SVIA or any anonymizer on overhead imagery was found on Lacuna. The only SVIA figure in the file is the 99.2% raw-image city re-identification on Cityscapes, and the post-SVIA value is not given. We will therefore establish this baseline ourselves.

A second strong competitor is dilated-mask inpainting, in which the mask is grown to cover the expected shadow and lean envelope for the maximum plausible height. A third is an inpainter trained end-to-end with an adversarial height-predictor penalty, with no explicit decoy.

Two ablations isolate the key ingredient. One removes shadow rendering or lean rendering from the defense. The other replaces shared multi-view decoys with independent per-view decoys.

Beating the baselines means the following, measured on the same held-out tiles. Attacker footprint IoU and height error against the true building must be statistically indistinguishable from a neighborhood-average guess, both single-view and cross-view. At the same time, downstream segmentation F1 on non-sensitive buildings must stay within a pre-registered margin of the unaltered-data model, and must beat SVIA-overhead and dilated masking at matched privacy. Privacy–utility curves are the primary deliverable.

Why not just…?

Why not just black out or blur the sensitive footprint? Destructive removal is exactly what failed for MRI. There it erased downstream signal (muscle radiodensity prediction lost significance, $p \le 10^{-4}$) and still left a structural prior that a diffusion model exploited (Pitfalls of defacing). In aerial images a black rectangle also announces "something sensitive is here," and it leaves the shadow and lean that sit outside the footprint untouched. We will measure both the utility cost and the residual height leak directly, so this remains a tested baseline rather than a dismissed one.

Why not just dilate the mask to cover shadows and lean, then use ordinary inpainting? This is the most serious threat. It may close most of the single-view leak. Its weaknesses are cost and brittleness. The envelope must assume a worst-case height, and at 54° off-nadir the lean of a tall building can sweep across neighboring structures, so dilation erases non-sensitive buildings and lowers utility. Nothing in dilation keeps views consistent either, so cross-view alignment may still recover the site. The matched-privacy utility comparison and the cross-view attack will settle this. If dilation matches our defense at equal utility, we will say so and recommend it.

Why not just train the inpainter against a height-estimator adversary, with no explicit geometry? An adversary only removes cues visible to that particular estimator, and the resulting images can carry geometrically impossible shadows. Those impossible shadows are themselves a detectable signature. We include this baseline and evaluate it against a held-out attacker with a different architecture. Our prediction is that explicit decoy rendering generalizes across attackers while adversarial training overfits its adversary. If the prediction fails, the explicit geometry is unnecessary, and we would drop it.

Why not just release only nadir images? Off-nadir collection is routine, and nadir-only models transfer poorly to off-nadir views (MVOI reports about a 65% loss in predictive power). Withholding angles is therefore a utility cost in its own right. It also leaves the shadow leak unaddressed.

Proposed Experiments

The first decisive check is the attack. On MVOI held-out tiles, compare attacker footprint IoU and height error for each sanitizer against the neighborhood-average guess, split by look-angle band and by whether an unsanitized cross-view image is available. The MRI study's gap between 0.38 mm (refaced) and 0.71 mm (average face) is the analogue of the separation we look for. If 2D inpainting shows no separation, the attack claim is refuted.

The second check is the privacy–utility curve. Train building segmentation on each sanitized release and report F1 on non-sensitive buildings per angle band, using the MVOI TernausNet figures (0.62 nadir, 0.22 beyond 40°) as the reference scale. Report image realism with FID, and run a detector that tries to tell sanitized sites from untouched ones.

The third check is scale and generality. We will repeat the vehicle-level version on DOTA, where Faster R-CNN with oriented boxes reaches 54.13% mAP. That experiment tests whether decoy sanitization of small objects preserves detection. Height leaks there should be weaker, which serves as an internal control. Finally, the ablations that remove shadow or lean rendering show which projected cue carries the leak. That result would be the most transferable finding for anyone releasing overhead data.