RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

Zilun Zhang, Tiancheng Zhao, Yulong Guo, Jianwei Yin

TL;DR

The authors introduce RS5M, the first large-scale remote sensing image-text dataset containing 5 million pairs, and GeoRSCLIP, a vision-language model fine-tuned on this data. By bridging the gap between general-purpose foundation models and specialized remote sensing tasks, GeoRSCLIP achieves significant performance gains in zero-shot classification, cross-modal retrieval, and semantic localization.

What Problem Does This Paper Address?

Vision-Language Models (VLMs) like CLIP have revolutionized computer vision by learning from massive amounts of internet data. However, these models are typically trained on "common objects" (dogs, cars, people), leading to a significant domain gap when applied to Remote Sensing (RS). Satellite and aerial imagery have unique characteristics: top-down perspectives, varying resolutions, and specialized content like specific land-cover types or industrial facilities.

Existing RS datasets are either small-scale with captions or large-scale with only simple class labels. This lack of massive, high-quality image-text pairs has been a bottleneck, preventing researchers from fully leveraging foundation models for tasks like zero-shot classification or natural language-based image retrieval in the RS domain.

Methods

The researchers propose a three-stage framework: GVLM (General VLM) $\rightarrow$ DVLM (Domain VLM) $\rightarrow$ DTM (Downstream Task Model). The core idea is to create a specialized intermediate model (the DVLM) that understands the RS domain before tackling specific tasks.

Fig. 1: Illustration of our proposed Framework. The Domain Vision-Language Model (DVLM) plays a central role in accepting the general knowledge from the General Vision-Language Model (GVLM) and is injected with massive domain-specific knowledge from external data. With the proper learning paradigm, DVLM is able to transfer the general knowledge with domain-specific prior to the Downstream Task Model (DTM) for domain-specific tasks. A demo for our proposed RS5M dataset is on the left.

To train this DVLM, they constructed RS5M through two main pipelines:

  1. Filtering (PUB11): They took 11 massive public image-text datasets (like LAION) and filtered them using RS-related keywords. They then used a pre-trained VLM and a custom RS image detector to remove non-RS outliers.
  2. Captioning (RS3): They took three large-scale RS datasets that only had class labels (MillionAID, FMoW, BigEarthNet) and used a tuned BLIP-2 model to generate descriptive captions. To ensure quality, they developed a rotation-invariant criterion: selecting captions that remained consistent even when the image was rotated—a crucial feature for satellite imagery.
Fig. 2: Overview of the collection process for RS5M. Circles represent different steps, gears stand for the model utilized, rectangles represent the images, and dash lines connect to the optional step.

Finally, they implemented GeoRSCLIP by fine-tuning CLIP on RS5M using various Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA and adapters, as well as full fine-tuning.

Datasets

  • RS5M (New): 5 million image-text pairs. It is nearly 1000 times larger than previous RS image-text paired datasets.
  • PUB11: A subset of RS5M (3M pairs) filtered from general web datasets.
  • RS3: A subset of RS5M (2M pairs) with machine-generated captions for existing RS image datasets.
  • Evaluation Benchmarks: The model was tested on standard RS datasets including AID, RESISC45, and EuroSAT (for classification); RSICD and RSITMD (for retrieval); and AIR-SLT (for semantic localization).

Key Insights and Ideas

  • Synthetic Captions are a Game Changer: By using a tuned BLIP-2 model to "describe" existing large-scale unlabeled or class-labeled RS images, the authors were able to scale their dataset far beyond what manual annotation would allow.
  • Domain Specialization is Essential: General models like CLIP underperform on RS data. The "DVLM" intermediate step effectively transfers general knowledge to the specialized RS domain.
  • Rotation Invariance Matters: Unlike natural images, satellite images don't have a fixed "up." Incorporating rotational invariance into caption selection helps the model learn more robust, orientation-independent features.
  • Scale and Diversity: Combining web-crawled "noisy" data with structured, machine-captioned domain data provides a balanced training signal that improves both generalization and domain-specific accuracy.

Key Results

GeoRSCLIP significantly outperforms the original CLIP baseline and previous state-of-the-art models across multiple tasks:

  • Zero-shot Classification: Improvements of 3% to 20% in top-1 accuracy on datasets like AID and EuroSAT.
  • Cross-Modal Retrieval: Gains of 3% to 6% in mean Recall for both image-to-text and text-to-image tasks.
  • Semantic Localization: A boost of 4% to 5% in the comprehensive $R_{mi}$ metric, showing a better ability to ground text descriptions in specific image regions.
  • Generative Capabilities: When used to tune Stable Diffusion (RS-SD), the model produced much more realistic satellite imagery compared to the base version, reducing the FID score from 36.86 to 28.32.

Limitations and Future Work

  • Geographic Bias: The dataset shows a long-tailed distribution in geolocation, with over-representation of North America and Europe and under-representation of regions like Middle Africa. This could introduce biases into the model.
  • Caption Noise: Despite rigorous filtering, machine-generated captions may still contain hallucinations or inaccuracies.
  • Interaction Modeling: Current PEFT methods don't fully account for the complex interactions between image and text modalities during fine-tuning.
  • Task Expansion: Future work aims to extend GeoRSCLIP to more complex RS tasks like change detection, object detection, and visual question answering (RSVQA).

Summary updated

Abstract

Pre-trained Vision-Language Models (VLMs) utilizing extensive image-text paired data have demonstrated unprecedented image-text association capabilities, achieving remarkable results across various downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. A critical challenge is how to make use of existing large-scale pre-trained VLMs, which are trained on common objects, to perform the domain-specific transfer for accomplishing domain-related downstream tasks. In this paper, we propose a new framework that includes the Domain pre-trained Vision-Language Model (DVLM), bridging the gap between the General Vision-Language Model (GVLM) and domain-specific downstream tasks. Moreover, we present an image-text paired dataset in the field of remote sensing (RS), RS5M, which has 5 million RS images with English descriptions. The dataset is obtained from filtering publicly available image-text paired datasets and captioning label-only RS datasets with pre-trained VLM. These constitute the first large-scale RS image-text paired dataset. Additionally, we fine-tuned the CLIP model and tried several Parameter-Efficient Fine-Tuning methods on RS5M to implement the DVLM. Experimental results show that our proposed dataset is highly effective for various tasks, and our model GeoRSCLIP improves upon the baseline or previous state-of-the-art model by $3\%\sim20\%$ in Zero-shot Classification (ZSC), $3\%\sim6\%$ in Remote Sensing Cross-Modal Text-Image Retrieval (RSCTIR) and $4\%\sim5\%$ in Semantic Localization (SeLo) tasks. Dataset and models have been released in: \url{https://github.com/om-ai-lab/RS5M}.
BibTeX
@article{zhang2024rs,
  author = {Zilun Zhang and Tiancheng Zhao and Yulong Guo and Jianwei Yin},
  title = {{RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2024},
  doi = {10.1109/tgrs.2024.3449154},
}
Figure