RemoteCLIP: A Vision Language Foundation Model for Remote Sensing

Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, Jun Zhou

Abstract

General-purpose foundation models have become increasingly important in the field of artificial intelligence. While self-supervised learning (SSL) and Masked Image Modeling (MIM) have led to promising results in building such foundation models for remote sensing, these models primarily learn low-level features, require annotated data for fine-tuning, and are not applicable for retrieval and zero-shot applications due to the lack of language understanding. In response to these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics, as well as aligned text embeddings for seamless downstream application. To address the scarcity of pre-training data, we leverage data scaling, converting heterogeneous annotations based on Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion, and further incorporating UAV imagery, resulting in a 12x larger pretraining dataset. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, k-NN classification, few-shot classification, image-text retrieval, and object counting. Evaluations on 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, show that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP outperforms previous SoTA by 9.14% mean recall on the RSITMD dataset and by 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the CLIP baseline by up to 6.39% average accuracy on 12 downstream datasets.

Introduction

While foundation models like CLIP and GPT have revolutionized general AI, the remote sensing (RS) community has primarily relied on Masked Image Modeling (MIM). MIM models, while effective at reconstruction, suffer from a significant semantic gap, learning low-level textures rather than high-level concepts. Furthermore, they lack a language component, precluding zero-shot capabilities.

A major bottleneck is data scarcity; high-quality image-caption pairs for satellite imagery are rare compared to natural image datasets. Additionally, general-purpose models like the original CLIP struggle with the unique "aerial view" of satellite imagery, which provides an unobstructed, top-down perspective fundamentally different from ground-level photos.

Methods: Data Scaling via Annotation Unification

The core innovation of RemoteCLIP is the unification of heterogeneous annotations to scale pre-training data. Since natural language captions are scarce in RS, the authors developed two key strategies to "recycle" existing specialized datasets:

  1. Box-to-Caption (B2C): A rule-based system that converts object detection labels (bounding boxes and class names) into natural language descriptions based on object counts and spatial locations.
  2. Mask-to-Box (M2B): A pre-processing step that converts semantic segmentation masks into bounding boxes, which are then processed by the B2C pipeline.
Overview of the RemoteCLIP data scaling pipeline through annotation unification.

By combining these converted datasets with UAV imagery and existing retrieval sets, the authors created a pre-training corpus 12 times larger than previous efforts. RemoteCLIP was then developed through continual pre-training on OpenAI’s original CLIP weights using the InfoNCE loss, aligning the model's existing language understanding with the specific visual domain of remote sensing.

Results and Discussion

RemoteCLIP achieves state-of-the-art (SOTA) performance across 16 downstream datasets. A key finding is that domain-specific alignment through continual pre-training is more impactful than raw model scale; for instance, a smaller RemoteCLIP (ResNet-50) can outperform a massive ViT-G model trained only on natural images.

Comparison of RemoteCLIP against previous SoTA and baseline CLIP models on the RET-3 benchmark.

In image-text retrieval, RemoteCLIP improved mean recall by 9.14% on RSITMD and 8.92% on RSICD. In zero-shot classification, it surpassed the vanilla CLIP baseline by an average of 6.39% accuracy. Furthermore, the newly introduced RemoteCount benchmark demonstrated that RemoteCLIP possesses a superior understanding of object quantities in aerial views compared to standard CLIP models.

Conclusion and Future Work

RemoteCLIP demonstrates that vision-language alignment is a powerful paradigm for remote sensing. While highly effective, limitations remain, such as performance on very low-resolution imagery and the relative simplicity of rule-based captions. Future work will explore the use of Large Language Models (LLMs) for more diverse caption generation, scaling to even larger vision transformers, and expanding the model's capabilities to multispectral and SAR data.

Summary updated

Abstract

General-purpose foundation models have led to recent breakthroughs in artificial intelligence (AI). In remote sensing, self-supervised learning (SSL) and masked image modeling (MIM) have been adopted to build foundation models. However, these models primarily learn low-level features and require annotated data for fine-tuning. Moreover, they are inapplicable for retrieval and zero-shot applications due to the lack of language understanding. To address these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics and aligned text embeddings for seamless downstream application. To address the scarcity of pretraining data, we leverage data scaling which converts heterogeneous annotations into a unified image-caption data format based on box-to-caption (B2C) and mask-to-box (M2B) conversion. By further incorporating unmanned aerial vehicle (UAV) imagery, we produce a $12\times $ larger pretraining dataset than the combination of all available datasets. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, k-NN classification, few-shot classification, image-text retrieval, and object counting in remote sensing images. Evaluation of 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, shows that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP beats the state-of-the-art (SOTA) method by 9.14% mean recall on the RSITMD dataset and 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the contrastive language image pretraining (CLIP) baseline by up to 6.39% average accuracy on 12 downstream datasets.
BibTeX
@article{liu2023remoteclip,
  author = {Fan Liu and Delong Chen and Zhangqingyun Guan and Xiaocong Zhou and Jiale Zhu and Qiaolin Ye and Liyong Fu and Jun Zhou},
  title = {{RemoteCLIP: A Vision Language Foundation Model for Remote Sensing}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2023},
  url = {https://openreview.net/forum?id=8h04C6yuya5},
}
Figure