TL;DR
Existing geospatial foundation models typically focus strictly on overhead satellite imagery, struggling to connect wide-area satellite views with ground-level street photography due to massive scale differences. GAIR bridges this divide by using an implicit neural representation module that smoothly interpolates satellite feature maps at arbitrary coordinates, enabling fine-grained, geo-aligned contrastive learning across satellite imagery, street view photos, and GPS metadata. The resulting model achieves state-of-the-art results across 9 distinct geospatial tasks spanning 22 datasets.
What Problem Does This Paper Address?
While vision foundation models have revolutionized machine learning, geospatial AI poses a unique challenge: real-world data spans radically disparate modalities and spatial scales. A single $256 \times 256$ Sentinel-2 satellite image might cover an entire neighborhood containing thousands of street view (SV) photos.
Standard Vision Transformers (ViTs) output fixed grid-patch tokens or global embeddings, meaning a street view image rarely aligns with an exact patch center in overhead imagery. Existing Geospatial Foundation Models (GeoFMs) either ignore ground-level imagery altogether or rely on coarse image-level objectives, failing to capture the localized geographic relationships necessary for fine-grained environmental reasoning, socio-economic analysis, and cross-modal geolocalization.
Methods
GAIR (Geo-Aligned Implicit Representations) uses a factorized architecture that aligns three modalities through self-supervised learning:
[Remote Sensing Image] ──► ViT Encoder ──► 2D Feature Map ──┐
├─► NILI (Local RS Embedding)
[Geolocation (Lat/Lon)] ─► RFF Location Encoder ─────────────┼───────────────┐
│ │
▼ ▼
[Street View Image] ──► ViT Encoder ──► SV Embedding ──► INCL Loss SECL Loss
- Factorized Encoders: GAIR uses separate ViT backbones for remote sensing (RS) imagery and street view (SV) images, alongside a Random Fourier Feature (RFF) location encoder for spherical GPS coordinates.
- Neural Implicit Local Interpolation (NILI): Inspired by continuous image super-resolution (e.g., LIIF), NILI acts as a local continuous decoder. Given query GPS coordinates within a satellite tile, NILI identifies the four nearest patch embeddings, computes continuous relative coordinate offsets, processes them through a shared MLP, and blends them using bilinear area weights into a continuous, localized RS feature vector.
- Dual Contrastive Objectives:
- Implicit Neural Contrastive Learning (INCL): Enforces bidirectional feature alignment between the localized RS embedding and the co-located street view photo embedding.
- Spatially Explicit Contrastive Learning (SECL): Leverages a memory bank of geographic coordinates to align explicit location embeddings with both the localized RS features and the SV images.
- Downstream Efficiency: During typical single-modality downstream tasks, the NILI module is discarded, incurring zero computational overhead at inference time.
Datasets
- Streetscapes1M (Pretraining): A newly curated multimodal dataset of 1 million tuples $(\text{RS image}, \text{SV image}, \text{location})$ sampled across 688 global cities from Global Streetscapes paired with multi-band Sentinel-2 imagery from Google Earth Engine.
- Urban-Rural Debiasing Set: 200,000 globally sampled, urban-rural balanced RS-location pairs used to mitigate geographical representation bias.
- Evaluation Benchmarks (22 datasets across 9 tasks):
- Street View Tasks: Socio-economic indicator regression (10 indicators in LA and Boston), human perception regression (6 perceptual scores), view direction classification, and imaging platform classification.
- Remote Sensing Tasks: HLS Burn Scars segmentation, AI4SmallFarms cropland boundary delineation, and South Sudan multi-temporal crop-type mapping.
- Location Tasks (LocBench / MOSAIKS): Species fine-grained classification (BirdSnap, NABirds, iNat2017, iNat2018) and satellite image regression (population density, forest cover, nighttime lights, elevation).
Key Insights and Ideas
- Continuous Localized Probing Beats Patch Pooling: Discarding NILI in favor of standard global pooling causes performance to drop markedly (e.g., crop segmentation drops from 55.53% to 55.01% mIoU, and socio-economic RMSE worsens from 0.8349 to 0.8588). Continuous interpolation is essential to resolve cross-scale geographic mismatches.
- Reconstruction Pretraining Fails Across Modalities: Training a masked autoencoder (GAIR-MAE) where street views reconstruct missing satellite patches resulted in unstable loss curves and significantly inferior transfer performance compared to contrastive alignment.
- Accurate Geolocation is a Structural Anchor: Injecting simulated GPS noise (via von Mises–Fisher perturbations) directly degrades downstream performance, confirming that precise coordinates provide vital geometric anchors for cross-view feature binding.
- Debiasing Without Performance Trade-Offs: Street view imagery heavily overrepresents dense urban cores. Continuing pretraining of the RS and location branches on globally balanced coordinates reduces the Geo-Bias Score (marked SSI) without degrading core downstream accuracy.
Key Results
- Remote Sensing Segmentation: Despite using only 0.1M satellite images for pretraining (far fewer than competing RS foundation models), GAIR achieves 83.26% mIoU on Burn Scars (+0.5% over Prithvi) and 43.35% mIoU on Cropland Boundary Segmentation (+16.16% over the best baseline).
- Street View Tasks: GAIR consistently outperforms unimodal MoCo-v3 and MAE models trained on the exact same dataset, dropping RMSE on socio-economic prediction from 0.6707 to 0.6612 and increasing view direction F1 from 0.5032 to 0.6102.
- Multimodal Fusion Gains: When combining all three modalities ($\text{RS} + \text{SV} + \text{Location}$) at inference time, GAIR delivers an additional 4% to 20% error reduction over single-modality baselines.
- Zero-Shot Geolocalization: GAIR outperforms GeoCLIP across all evaluation radii on 15,000 queries (achieving 10.04% vs. 9.45% accuracy within 1 km and 33.15% vs. 30.11% within 200 km).
Limitations and Future Work
- View Direction Sensitivity: GAIR currently ignores camera orientation (camera azimuth/heading), treating ground-level photos as isotropic points on the satellite feature map. Incorporating viewing frustums or compass headings is an important next step.
- Static Input Resolutions: Model inputs are currently fixed at $96 \times 96$ for satellite crops and $224 \times 224$ for street view photos. Future extensions should accommodate variable spatial resolutions and field-of-view footprints.
Summary updated