GAIR: A Multimodal Geo-Foundation Model with Geo-Aligned Implicit Representations

Zeping Liu, Zhangyu Wang, Junfeng Jiao, Ni Lao, Gengchen MaiSubmission to ICLR 2026 · Withdrawn

TL;DR

Existing geospatial foundation models typically focus strictly on overhead satellite imagery, struggling to connect wide-area satellite views with ground-level street photography due to massive scale differences. GAIR bridges this divide by using an implicit neural representation module that smoothly interpolates satellite feature maps at arbitrary coordinates, enabling fine-grained, geo-aligned contrastive learning across satellite imagery, street view photos, and GPS metadata. The resulting model achieves state-of-the-art results across 9 distinct geospatial tasks spanning 22 datasets.

What Problem Does This Paper Address?

While vision foundation models have revolutionized machine learning, geospatial AI poses a unique challenge: real-world data spans radically disparate modalities and spatial scales. A single $256 \times 256$ Sentinel-2 satellite image might cover an entire neighborhood containing thousands of street view (SV) photos.

Standard Vision Transformers (ViTs) output fixed grid-patch tokens or global embeddings, meaning a street view image rarely aligns with an exact patch center in overhead imagery. Existing Geospatial Foundation Models (GeoFMs) either ignore ground-level imagery altogether or rely on coarse image-level objectives, failing to capture the localized geographic relationships necessary for fine-grained environmental reasoning, socio-economic analysis, and cross-modal geolocalization.

Methods

GAIR (Geo-Aligned Implicit Representations) uses a factorized architecture that aligns three modalities through self-supervised learning:

[Remote Sensing Image] ──► ViT Encoder ──► 2D Feature Map ──┐
                                                             ├─► NILI (Local RS Embedding)
[Geolocation (Lat/Lon)] ─► RFF Location Encoder ─────────────┼───────────────┐
                                                             │               │
                                                             ▼               ▼
[Street View Image]   ──► ViT Encoder ──► SV Embedding ──► INCL Loss    SECL Loss
  • Factorized Encoders: GAIR uses separate ViT backbones for remote sensing (RS) imagery and street view (SV) images, alongside a Random Fourier Feature (RFF) location encoder for spherical GPS coordinates.
  • Neural Implicit Local Interpolation (NILI): Inspired by continuous image super-resolution (e.g., LIIF), NILI acts as a local continuous decoder. Given query GPS coordinates within a satellite tile, NILI identifies the four nearest patch embeddings, computes continuous relative coordinate offsets, processes them through a shared MLP, and blends them using bilinear area weights into a continuous, localized RS feature vector.
  • Dual Contrastive Objectives:
  • Implicit Neural Contrastive Learning (INCL): Enforces bidirectional feature alignment between the localized RS embedding and the co-located street view photo embedding.
  • Spatially Explicit Contrastive Learning (SECL): Leverages a memory bank of geographic coordinates to align explicit location embeddings with both the localized RS features and the SV images.
  • Downstream Efficiency: During typical single-modality downstream tasks, the NILI module is discarded, incurring zero computational overhead at inference time.

Datasets

  • Streetscapes1M (Pretraining): A newly curated multimodal dataset of 1 million tuples $(\text{RS image}, \text{SV image}, \text{location})$ sampled across 688 global cities from Global Streetscapes paired with multi-band Sentinel-2 imagery from Google Earth Engine.
  • Urban-Rural Debiasing Set: 200,000 globally sampled, urban-rural balanced RS-location pairs used to mitigate geographical representation bias.
  • Evaluation Benchmarks (22 datasets across 9 tasks):
  • Street View Tasks: Socio-economic indicator regression (10 indicators in LA and Boston), human perception regression (6 perceptual scores), view direction classification, and imaging platform classification.
  • Remote Sensing Tasks: HLS Burn Scars segmentation, AI4SmallFarms cropland boundary delineation, and South Sudan multi-temporal crop-type mapping.
  • Location Tasks (LocBench / MOSAIKS): Species fine-grained classification (BirdSnap, NABirds, iNat2017, iNat2018) and satellite image regression (population density, forest cover, nighttime lights, elevation).

Key Insights and Ideas

  • Continuous Localized Probing Beats Patch Pooling: Discarding NILI in favor of standard global pooling causes performance to drop markedly (e.g., crop segmentation drops from 55.53% to 55.01% mIoU, and socio-economic RMSE worsens from 0.8349 to 0.8588). Continuous interpolation is essential to resolve cross-scale geographic mismatches.
  • Reconstruction Pretraining Fails Across Modalities: Training a masked autoencoder (GAIR-MAE) where street views reconstruct missing satellite patches resulted in unstable loss curves and significantly inferior transfer performance compared to contrastive alignment.
  • Accurate Geolocation is a Structural Anchor: Injecting simulated GPS noise (via von Mises–Fisher perturbations) directly degrades downstream performance, confirming that precise coordinates provide vital geometric anchors for cross-view feature binding.
  • Debiasing Without Performance Trade-Offs: Street view imagery heavily overrepresents dense urban cores. Continuing pretraining of the RS and location branches on globally balanced coordinates reduces the Geo-Bias Score (marked SSI) without degrading core downstream accuracy.

Key Results

  • Remote Sensing Segmentation: Despite using only 0.1M satellite images for pretraining (far fewer than competing RS foundation models), GAIR achieves 83.26% mIoU on Burn Scars (+0.5% over Prithvi) and 43.35% mIoU on Cropland Boundary Segmentation (+16.16% over the best baseline).
  • Street View Tasks: GAIR consistently outperforms unimodal MoCo-v3 and MAE models trained on the exact same dataset, dropping RMSE on socio-economic prediction from 0.6707 to 0.6612 and increasing view direction F1 from 0.5032 to 0.6102.
  • Multimodal Fusion Gains: When combining all three modalities ($\text{RS} + \text{SV} + \text{Location}$) at inference time, GAIR delivers an additional 4% to 20% error reduction over single-modality baselines.
  • Zero-Shot Geolocalization: GAIR outperforms GeoCLIP across all evaluation radii on 15,000 queries (achieving 10.04% vs. 9.45% accuracy within 1 km and 33.15% vs. 30.11% within 200 km).

Limitations and Future Work

  • View Direction Sensitivity: GAIR currently ignores camera orientation (camera azimuth/heading), treating ground-level photos as isotropic points on the satellite feature map. Incorporating viewing frustums or compass headings is an important next step.
  • Static Input Resolutions: Model inputs are currently fixed at $96 \times 96$ for satellite crops and $224 \times 224$ for street view photos. Future extensions should accommodate variable spatial resolutions and field-of-view footprints.

Summary updated

Abstract

Vision Transformer (ViT) has been used in many computer vision tasks with excellent results by providing representations for a whole image or image patches. However, ViT lacks detailed localized image representations when applied to geospatial tasks that involve multiple geospatial data modalities, such as overhead remote sensing (RS) data, ground-level imagery, and geospatial vector data. Here high-resolution localized representations are vital for modeling geospatial relationships and alignments across modalities. We proposed to solve this representation problem with an implicit neural representation (INR) module extending ViT with Neural Implicit Local Interpolation, which produces a continuous RS image representation covering arbitrary location in the RS image. Based on the INR module, we propose GAIR, a multimodal Geo-Foundation Model (GeoFM) integrating overhead RS data, street view (SV) imagery, and their geolocation metadata. GAIR utilizes three factorized neural encoders to project different modalities into the embedding space, and the INR module is used to further align these representations geographically, which are trained with contrastive learning objectives from unlabeled data. We evaluate GAIR across 9 geospatial tasks and 22 datasets spanning RS image-based, SV image-based, and location embedding-based benchmarks. Experimental results demonstrate that GAIR outperforms state-of-the-art GeoFMs and alternative training objectives (e.g., MoCo-V2 and MAE) that do not use fine-grained geo-aligned spatial representations. Our results highlight the effectiveness of GAIR in learning generalizable geospatial representations across tasks, spatial scales, and temporal contexts.
BibTeX
@misc{liu2025gair,
  author = {Zeping Liu and Zhangyu Wang and Junfeng Jiao and Ni Lao and Gengchen Mai},
  title = {{GAIR: A Multimodal Geo-Foundation Model with Geo-Aligned Implicit Representations}},
  year = {2025},
  eprint = {2503.16683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2503.16683},
}
Figure