When a satellite photographs a city from an angle instead of straight down, tall buildings appear to lean, their walls come into view, and shadows fall differently. Segmentation models trained on near-vertical images break down under these conditions. On the SpaceNet multi-view benchmark, a strong building segmenter falls from an F1 of 0.62 at nadir (looking straight down) to 0.22 beyond 40° off-nadir (SpaceNet MVOI). Face recognition has dealt with a structurally identical problem for years. A person's photos taken from different head angles can look less alike than photos of two different people taken from the same angle. Face recognition handles this "intra-personal variation" by supervising with identity: many photos of the same person are grouped together, and the model learns which feature directions carry identity and which carry nuisance variation such as pose or lighting.
We propose to apply that recipe to overhead imagery. SpaceNet MVOI photographs the same 126,747 Atlanta buildings from 27 look angles in one satellite pass. Each building is therefore an "identity" observed under many real, physically produced viewpoints. We pretrain an image encoder so that the features of one building agree across look angles. We then estimate the subspace of feature directions along which the same building varies with viewpoint, and remove that subspace at test time without needing any acquisition metadata. The central test is whether this generalizes to look angles never seen in training, in particular across the north/south asymmetry that MVOI documents. If viewpoint variation in overhead imagery is low-dimensional and structured (lean direction follows sensor azimuth, lean magnitude grows with off-nadir angle), then a variation subspace learned from one side of the sky should transfer to the other. If it does not, we learn that the problem is fundamentally one of coverage rather than representation.
Core Claim
Real multi-view revisits of the same physical object are a source of identity supervision that remote sensing has not used. Seasonal Contrast (SeCo) is the closest precedent. It treats repeat Sentinel-2 captures of the same place across seasons as positive pairs, the image pairs a contrastive loss pulls together, and it shows that this beats ImageNet initialization (BigEarthNet mAP 87.81 vs. 86.74; OSCD change-detection F1 46.94 vs. 36.20). SeCo's revisits are all near-nadir at 10 m resolution, so it never sees lean or wall exposure and cannot separate individual objects.
We change the variation axis from time to viewing geometry and the unit of identity from a scene tile to a single building. We also add an explicit decomposition of each feature vector into identity and viewpoint parts. The decomposition is modeled on the Joint Bayesian formulation evaluated in Hu et al., where separating identity covariance from intra-personal covariance raised LFW verification from 83.33% to 87.63% over cosine matching on the same fused features. The claim is falsifiable: an encoder trained this way on nadir plus one-sided off-nadir looks should close a substantial part of the F1 gap on held-out opposite-side and very-off-nadir looks, compared with the same encoder trained with synthetic augmentation alone.
How This Differs from the Off-Nadir Literature
Two existing lines of work attack off-nadir degradation, and both need something this proposal avoids. NaGAN uses an adversarial network to warp off-nadir detector features toward a "nadir-like" representation, with the warp conditioned on look-angle metadata, and reports [email protected] gains of 1.4 points on all angles and 1.6 points on off-nadir angles. Hao et al. inject off-nadir angle and ground sampling distance through affine modulation layers in a U-Net and add uncertainty modeling, reaching F1 0.743 against a 0.712 U-Net baseline.
Both require calibrated metadata at inference, and both were trained with all angles available. Neither tests extrapolation to unseen angles. NaGAN also forces every view toward nadir, which risks discarding the wall and profile cues that MVOI suggests broader-prior models retain.
Our method uses metadata only during training, and only to decide which crops belong to the same building. Test-time inputs are raw pixels. The same identity framing also makes available a second evaluation that neither line addresses: re-identifying a building across looks. VRAI does instance re-identification from the air, for vehicles seen by drones, reaching 80.30% Rank-1 against roughly 97% for humans. Its views are low-altitude and near-nadir, and its part-detector design depends on rooftop landmarks that off-nadir geometry deforms.
CLIP-Cluster motivates a later step. It closes fragmented face clusters by shifting features along text-defined attribute directions such as age or pose. Web text contains no notion of "look angle 54°", so we estimate the variation directions from real revisits instead of from prompts.
Research Program
The encoder is a ResNet-50 with a feature pyramid, the standard choice in this literature and the backbone NaGAN uses. Pretraining uses only MVOI images from the training angles. For each building, we pool features from a context-padded crop around its footprint in every training-angle image. The padding is needed because MVOI footprints are defined on nadir geometry, and in oblique views the roof shifts away from the footprint. A contrastive loss treats the crops of one building from different angles as positives and other buildings as negatives, and it is applied jointly with a standard tile-level augmentation loss so that background context is also learned.
After pretraining, we estimate two covariance matrices over the pooled building features. The within-building covariance $S_w$ measures how one building's features spread across look angles. The between-building covariance $S_b$ measures how different buildings' average features spread. At test time, features are projected away from the top $k$ eigenvectors of $S_w$ that carry little between-building variance. This is a light, metadata-free analogue of Joint Bayesian and subspace methods from face verification.
As a diagnostic, we also train a linear probe that predicts look angle and azimuth from the removed component. A clean decomposition should put the geometry information there and leave little of it in the retained component. The pretrained encoder is then fine-tuned for building segmentation, with labels only from training angles, and evaluated across all angle bins.
The analysis question that decides the whole program is extrapolation. We will measure what fraction of the within-building variance on held-out angles is captured by the top-$k$ directions estimated from training angles. If that fraction is high, the variation is structured and the method should transfer to unseen angles. If it is low, especially for the opposite side, then MVOI's asymmetry reflects genuinely new appearance (different shadow and wall configurations) that no subspace learned on one side can anticipate. That would be a reportable negative result, and it would point toward adding views rather than learning invariances.
Baselines
The naive reference point is an ImageNet-initialized network of the same architecture, trained for segmentation on the training angles with standard flips, rotations, scaling, and color jitter. This is how MVOI's own baselines behave. The cited number for this regime is TernausNet's F1 of 0.62 at nadir and 0.22 beyond 40° (SpaceNet MVOI). That model was trained with angles available, so it is not a held-out-angle number. For our held-out protocol, no reported result found on Lacuna, and we will report the naive baseline ourselves.
The strongest existing competitor is the metadata-conditioned segmenter of Hao et al., reported at F1 0.743 overall on SpaceNet 4. It is the strongest because it uses the very angle information our method withholds at test time. We will re-implement it with the same backbone, the same training angles, and the same tuning budget, and give it oracle metadata at test. Beating it means matching or exceeding its held-out-angle F1 without metadata. Matching it is itself a meaningful result, because metadata is routinely missing in practice.
We also include SeCo's released pretrained weights fine-tuned the same way, which represents temporal-revisit pretraining. A NaGAN-style adversarial nadir-alignment loss is attached to our segmentation decoder to represent canonical normalization.
Two ablations isolate the key ingredient. In the first, the contrastive positives come only from synthetic augmentations of a single view. Everything else stays the same: images, compute, and number of pretraining steps. This removes the real multi-view signal. The second keeps the multi-view pretraining but skips the subspace removal at test time, which shows whether the decomposition adds anything beyond the pretraining. Every arm gets the same hyperparameter search budget, measured in trials, on a validation split drawn from the training angles only.
Why not just…?
Why not just train on more angles with heavier augmentation? If every angle is available, this is probably sufficient. Our protocol deliberately withholds angles because real archives over most regions cover few look angles. MVOI's evidence suggests augmentation cannot replace the missing views. Its downsampling control shows that the collapse comes from lean and wall exposure rather than lower resolution, and planar affine transforms do not synthesize either. The augmentation-only ablation tests this directly on our split.
Why not just condition on metadata, as Hao et al. or NaGAN do? Conditioning is a genuine threat when metadata is clean, and the oracle-metadata baseline is included for exactly that reason. Our argument is not that conditioning is worse. Our method needs no metadata at inference, and conditioning methods have not been shown to extrapolate to angles outside their training range. If oracle conditioning dominates us on held-out angles, the proposal reduces to a fallback for missing metadata, and we will say so.
Why not just use generic self-supervised pretraining such as SeCo or MoCo? SeCo's gains are real, but its revisits never vary viewing geometry, so nothing in its objective asks for invariance to lean. The single-view contrastive ablation and the SeCo-weights baseline together measure how much of any improvement comes from in-domain self-supervision in general, as opposed to multi-view identity supervision specifically.
Why not just use tile-level multi-view contrast instead of per-building identities? This is the most serious simplification. It is SeCo with angle replacing season. It needs no footprint crops and could plausibly capture most of the benefit. We will run it as a fourth arm. If it matches building-level contrast on segmentation, we adopt it as the method. The building-level version is then justified only if it wins on cross-angle building re-identification, where instance discrimination should matter.
Proposed Experiments
The primary benchmark is SpaceNet MVOI/SpaceNet 4. Training angles are nadir and moderately off-nadir looks from one side. Evaluation uses all 27 angles, binned by MVOI's nadir, off-nadir, and very-off-nadir groups and split by side. Metrics are building F1 per bin and MVOI's Generalization Score, which normalizes retention at unseen angles by task difficulty. We run three seeds per arm and report confidence intervals.
The second evaluation is cross-look building re-identification, built from the same data. Queries are crops of a building at held-out angles, the gallery is crops at training angles, and the gallery includes nearby buildings as distractors. We report Rank-1 and mAP. No reported result found on Lacuna for this task.
A transfer check fine-tunes the pretrained encoders on SpaceNet 7. Its multi-temporal tracking baseline is reported near 0.40 (Van Etten et al.), and the check probes whether the representation is more than Atlanta-specific. Because SpaceNet 7 is near-nadir and globally distributed, it tests whether we damage general features. It does not test view invariance.
Three outcomes would decide the proposal. First, the gap between training-side and held-out-side F1 should shrink relative to both the augmentation-only and SeCo arms. We would consider a reduction of at least a quarter of the naive gap convincing. Second, training-angle within-building directions should explain most of the held-out within-building variance. If this holds while the segmentation gap persists, the bottleneck is the decoder rather than the representation. If neither holds, the asymmetry is new information and the method fails for a reason we can name. Third, the removed feature component should predict look angle while the retained component should not. A finding that only metadata-conditioned models extrapolate would also be informative: it would show that viewpoint in overhead imagery is better treated as an input than as a nuisance to be removed, unlike pose in face recognition.
References
- SpaceNet MVOI: A Multi-View Overhead Imagery Dataset
- Seasonal Contrast: Unsupervised Pre-Training from Uncurated Remote Sensing Data
- When Face Recognition Meets with Deep Learning: An Evaluation of Convolutional Neural Networks for Face Recognition
- NaGAN: Nadir-like Generative Adversarial Network for Off-Nadir Object Detection of Multi-View Remote Sensing Imagery
- Improving Building Segmentation for Off-Nadir Satellite Imagery
- Vehicle Re-Identification in Aerial Imagery: Dataset and Approach
- CLIP-Cluster: CLIP-Guided Attribute Hallucination for Face Clustering
- The Multi-Temporal Urban Development SpaceNet Dataset