Sky2Ground: A Benchmark for Site Modeling under Varying Altitude

Zengyan Wang, Sirshapan Mitra, Rajat C. Modi, Grace Lim, Yogesh Singh Rawat

Abstract

We introduce Sky2Ground, a three-view dataset designed for varying altitude camera localization, correspondence learning, and reconstruction. The dataset combines structured synthetic imagery with real, in-the-wild images, providing both controlled multi-view geometry and realistic scene noise. Each of the 51 sites contains thousands of satellite, aerial, and ground images spanning wide altitude ranges and nearly orthogonal viewing angles, enabling rigorous evaluation across global-to-local contexts. We benchmark state of the art pose estimation models, including MASt3R, DUSt3R, Map Anything, and VGGT, and observe that the use of satellite imagery often degrades performance, highlighting the challenges under large altitude variations. We also examine reconstruction methods, highlighting the challenges introduced by sparse geometric overlap, varying perspectives, and the use of real imagery, which often introduces noise and reduces rendering quality. To address some of these challenges, we propose SkyNet, a model which enhances cross-view consistency when incorporating satellite imagery with a curriculum-based training strategy to progressively incorporate more satellite views. SkyNet significantly strengthens multi-view alignment and outperforms existing methods by 9.6% on RRA@5 and 18.1% on RTA@5 in terms of absolute performance. Sky2Ground and SkyNet together establish a comprehensive testbed and baseline for advancing large-scale, multi-altitude 3D perception and generalizable camera localization. Code and models will be released publicly for future research.

Introduction: Bridging the Altitude Gap

In computer vision, site modeling has traditionally been confined to single-scale perspectives, such as drone-based city reconstruction or ground-level image localization. However, true planet-wide reconstruction requires a unified approach that bridges the massive altitude gap between satellite views (kilometers up), aerial views (hundreds of meters up), and ground-level imagery (eye level).

The primary challenge lies in the "distribution shift" caused by nearly orthogonal viewing angles and extreme variations in resolution and appearance across these tiers. Current state-of-the-art models often see their performance degrade when satellite imagery is introduced, as the features visible from space rarely align intuitively with those on the street, leading to unstable training and poor 3D alignment.

The Sky2Ground Dataset

To facilitate research in multi-altitude perception, we introduce the Sky2Ground dataset. It comprises 51 unique urban sites (e.g., Westminster Pier, Pyramide du Louvre), totaling 80,000 images. The dataset is categorized into three distinct altitude tiers: Satellite (1–2 km), Aerial (250–800 m), and Ground (5–80 m).

Overview of the Sky2Ground dataset, including camera trajectories and geographic distribution.

Sky2Ground uniquely combines high-fidelity synthetic data from Google Earth—complete with dense depth maps and precise camera intrinsics—with "real-world" imagery scraped from Google Maps reviews and YouTube travel vlogs. This hybrid approach provides both a controlled environment for geometric learning and a robust test for sim-to-real transfer under realistic scene noise.

Methodology: SkyNet and Curriculum Learning

We propose SkyNet, a two-stream architecture designed to handle the integration of satellite imagery without polluting high-resolution features. SkyNet utilizes a Ground-Aerial-Satellite (GAS) encoder alongside a dedicated Sat-Encoder. A key innovation is the Masked-Satellite-Attention (MSA) module, which restricts information flow to prevent satellite noise from destabilizing the learning of ground and aerial features, while still allowing satellite views to provide essential global alignment context.

Training is conducted via a two-pronged curriculum strategy:

  1. Curriculum-Aware Camera-Sampling (CA-CS): The model first learns to match cameras that are physically close before moving to non-overlapping, distant views.
  2. Progressive-View Sampling (P-VS): Using aerial imagery as a "connective tissue," training starts with aerial-heavy batches and gradually transitions to the more difficult ground-to-satellite pairs.

Key Results and Benchmarking

Our evaluation reveals a "Satellite Paradox": for many existing models, the introduction of satellite imagery actually hurts performance. The visual difference is so jarring that it destabilizes single-feedforward networks. SkyNet overcomes this, setting a new high-water mark for cross-view localization.

Performance comparison of baseline models across different data splits in the Sky2Ground benchmark.

On the full GAS setup, SkyNet achieved 83.2% RRA@5° (Relative Rotation Accuracy) and 66.5% RTA@5m (Relative Translation Accuracy), outperforming the previous best model, VGGT, by 9.6% and 18.1% respectively. Additionally, we found that 2D Gaussian Splatting (2DGS) consistently provided better perceptual quality and sharper contours than 3DGS when dealing with the sparse geometric overlap inherent in multi-altitude datasets.

Conclusion

Sky2Ground and SkyNet establish a comprehensive testbed for advancing large-scale, multi-altitude 3D perception. While SkyNet significantly strengthens multi-view alignment, future work will focus on developing unified models capable of predicting both camera poses and Gaussian parameters simultaneously to further improve rendering quality in real-world scenarios.

Summary updated

Abstract

We introduce Sky2Ground, a three-view dataset designed for varying altitude camera localization, correspondence learning, and reconstruction. The dataset combines structured synthetic imagery with real, in-the-wild images, providing both controlled multi-view geometry and realistic scene noise. Each of the 51 sites contains thousands of satellite, aerial, and ground images spanning wide altitude ranges and nearly orthogonal viewing angles, enabling rigorous evaluation across global-to-local contexts. We benchmark state of the art pose estimation models, including MASt3R, DUSt3R, Map Anything, and VGGT, and observe that the use of satellite imagery often degrades performance, highlighting the challenges under large altitude variations. We also examine reconstruction methods, highlighting the challenges introduced by sparse geometric overlap, varying perspectives, and the use of real imagery, which often introduces noise and reduces rendering quality. To address some of these challenges, we propose SkyNet, a model which enhances cross-view consistency when incorporating satellite imagery with a curriculum-based training strategy to progressively incorporate more satellite views. SkyNet significantly strengthens multi-view alignment and outperforms existing methods by 9.6% on RRA@5 and 18.1% on RTA@5 in terms of absolute performance. Sky2Ground and SkyNet together establish a comprehensive testbed and baseline for advancing large-scale, multi-altitude 3D perception and generalizable camera localization. Code and models will be released publicly for future research.
BibTeX
@inproceedings{wang2026sky,
  author = {Zengyan Wang and Sirshapan Mitra and Rajat C. Modi and Grace Lim and Yogesh Singh Rawat},
  title = {{Sky2Ground: A Benchmark for Site Modeling under Varying Altitude}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2026},
  url = {https://openreview.net/forum?id=NSH0WLObwQ},
}
Figure