Vision Foundation Models in Remote Sensing: A survey

Siqi Lu, Junlin Guo, James Zimmer‐Dauphinee, Jordan M. Nieusma, Xiao Wang, Parker VanValkenburgh, Steven A. Wernke, Yuankai Huo

TL;DR

This paper surveys the rapidly evolving field of vision foundation models (FMs) for remote sensing (RS), categorizing them by architecture, pre-training data, and methods. It highlights how self-supervised learning (SSL) and transformer architectures are driving significant performance gains across diverse RS tasks, from environmental monitoring to disaster management.

What Problem Does This Paper Address?

Remote sensing (RS) traditionally relied on manual interpretation and task-specific models, which were data-hungry and computationally expensive. The advent of large-scale, pre-trained "foundation models" (FMs) offers a paradigm shift, enabling a wide array of tasks with unprecedented accuracy and efficiency. However, the sheer volume of new FMs and their diverse applications in RS can be overwhelming for researchers. This paper aims to provide a structured overview of these recent FMs, their underlying methodologies, and their performance across various RS tasks, serving as a crucial resource for the community.

Methods

The paper categorizes FMs based on their architecture (e.g., CNNs, Vision Transformers), pre-training datasets, and pre-training methodologies. A key focus is on self-supervised learning (SSL) techniques, particularly contrastive learning (CL) and masked image modeling (MIM), including masked autoencoders (MAEs). These methods allow models to learn rich representations from vast amounts of unlabeled RS data, reducing reliance on expensive manual annotations. The survey also discusses the role of Vision Transformers (ViTs) in capturing long-range dependencies crucial for high-resolution RS imagery.

Datasets

The paper references a variety of datasets used for training and evaluating FMs in RS. These include:

  • ImageNet: A general-purpose image dataset often used for initial pre-training.
  • BigEarthNet: A large-scale benchmark for image-level classification in RS.
  • Sentinel-1, Sentinel-2, Landsat, NAIP, Gaofen-1/2/6: Various satellite imagery sources providing multispectral, SAR, and high-resolution data.
  • fMoW (Functional Map of the World): A dataset for scene classification and object detection.
  • DOTA (Dataset for Object Detection in Aerial Images): A large-scale dataset specifically for object detection in aerial imagery.
  • ISPRS Potsdam: A dataset for pixel-level tasks like semantic segmentation.
  • OSCD (Onera Satellite Change Detection) and LEVIR-CD: Datasets for change detection tasks.

The paper also notes the increasing importance of multimodal datasets that combine different sensor types (e.g., optical and SAR).

Key Insights and Ideas

  • SSL is a Game-Changer: Self-supervised learning, especially contrastive learning and masked autoencoders, is revolutionizing RS by enabling models to learn powerful representations from abundant unlabeled data, significantly reducing the need for costly manual annotations.
  • Transformers Conquer Long-Range Dependencies: Vision Transformers (ViTs) are proving highly effective in RS due to their ability to model long-range spatial and temporal dependencies, which are critical for analyzing high-resolution satellite imagery.
  • Multi-Level Analysis is Key: FMs are enabling sophisticated analysis at image-level (scene classification), region-level (object detection), and pixel-level (semantic segmentation, change detection), catering to diverse application needs.
  • Pre-training Matters Immensely: The choice of pre-training method and dataset has a profound impact on model performance, with SSL techniques consistently outperforming traditional supervised methods, especially in data-scarce scenarios.
  • Domain Adaptation is Crucial: While FMs offer generalization, adapting them to specific RS domains and tasks remains an active area of research, with challenges in computational resources and data quality.

Key Results

The survey highlights significant performance improvements across various RS tasks due to FMs. While specific quantitative results are presented in tables, key takeaways include:

  • Image-Level Classification: Models like msGFM and SkySense achieved high mAP scores (e.g., 92.90% and 92.09% on BigEarthNet), demonstrating strong capabilities in classifying entire scenes.
  • Pixel-Level Segmentation: Models like SkySense and CMID showed top performance on the ISPRS Potsdam dataset, with high mF1 and mIoU scores, respectively.
  • Region-Level Object Detection: RVSA achieved the highest mAP on the DOTA dataset (81.24%), showcasing strong object detection abilities.
  • Change Detection: SkySense and GFM demonstrated strong F1 scores on datasets like OSCD, while MTP and SkySense excelled on LEVIR-CD.
  • Pre-training Impact: Models pre-trained with SSL techniques like contrastive learning and MAE consistently showed superior performance compared to those trained with traditional supervised learning.

Limitations and Future Work

Limitations:

  • Scope and Coverage: The survey focuses on models released between June 2021 and June 2024, acknowledging that some recent advancements might be omitted due to release timing or lack of thorough evaluation.
  • Limited Testing: Many FMs have only been tested on a specific set of downstream applications, suggesting their potential for broader applicability is yet to be fully explored.
  • Evolving Field: The rapid pace of AI and RS research means the field is constantly evolving, making any survey a snapshot in time.

Future Work:

  • Efficient Model Development: Research into techniques like model distillation, pruning, and quantization to reduce computational requirements without compromising performance.
  • Multi-Modal Data Integration: Developing more sophisticated methods for integrating diverse data sources (e.g., optical, radar, DEM) to achieve more comprehensive insights.
  • Interdisciplinary Collaboration: Fostering partnerships between AI researchers, remote sensing experts, and domain specialists to address complex challenges and drive innovation.
  • Dataset Diversity and Quality: Addressing the need for more diverse, high-quality, and well-annotated datasets to further improve model generalization and robustness.
  • Scalability and Deployment: Developing models that are not only accurate but also scalable and efficient for real-time deployment in resource-constrained environments.

Summary updated

Abstract

Artificial intelligence (AI) technologies have profoundly transformed the field of remote sensing (RS), revolutionizing data collection, processing, and analysis. Traditionally reliant on manual interpretation and task-specific models, RS research has been significantly enhanced by the advent of foundation models (FMs)—large-scale pretrained AI models capable of performing a wide array of tasks with unprecedented accuracy and efficiency. This article provides a comprehensive survey of FMs in the RS domain. We categorize these models based on their architectures, pretraining datasets, and methodologies. Through detailed performance comparisons, we highlight emerging trends and the significant advancements achieved by those FMs. Additionally, we discuss technical challenges, practical implications, and future research directions, addressing the need for high-quality data, computational resources, and improved model generalization. Our research also finds that pretraining methods, particularly self-supervised learning (SSL) techniques like contrastive learning (CL) and masked autoencoders (MAEs), remarkably enhance the performance and robustness of FMs. This survey aims to serve as a resource for researchers and practitioners by providing a panorama of advances and promising pathways for the continued development and application of FMs in RS.
BibTeX
@article{lu2025vision,
  author = {Siqi Lu and Junlin Guo and James Zimmer‐Dauphinee and Jordan M. Nieusma and Xiao Wang and Parker VanValkenburgh and Steven A. Wernke and Yuankai Huo},
  title = {{Vision Foundation Models in Remote Sensing: A survey}},
  journal = {IEEE Geoscience and Remote Sensing Magazine},
  year = {2025},
  doi = {10.1109/mgrs.2025.3541952},
}

Loading related content...

Figure