Abstract
General-purpose foundation models have become increasingly important in the field of artificial intelligence. While self-supervised learning (SSL) and Masked Image Modeling (MIM) have led to promising results in building such foundation models for remote sensing, these models primarily learn low-level features, require annotated data for fine-tuning, and are not applicable for retrieval and zero-shot applications due to the lack of language understanding. In response to these limitations, we propose RemoteCLIP, the first vision-language foundation model for remote sensing that aims to learn robust visual features with rich semantics, as well as aligned text embeddings for seamless downstream application. To address the scarcity of pre-training data, we leverage data scaling, converting heterogeneous annotations based on Box-to-Caption (B2C) and Mask-to-Box (M2B) conversion, and further incorporating UAV imagery, resulting in a 12x larger pretraining dataset. RemoteCLIP can be applied to a variety of downstream tasks, including zero-shot image classification, linear probing, k-NN classification, few-shot classification, image-text retrieval, and object counting. Evaluations on 16 datasets, including a newly introduced RemoteCount benchmark to test the object counting ability, show that RemoteCLIP consistently outperforms baseline foundation models across different model scales. Impressively, RemoteCLIP outperforms previous SoTA by 9.14% mean recall on the RSITMD dataset and by 8.92% on the RSICD dataset. For zero-shot classification, our RemoteCLIP outperforms the CLIP baseline by up to 6.39% average accuracy on 12 downstream datasets.
Introduction
While foundation models like CLIP and GPT have revolutionized general AI, the remote sensing (RS) community has primarily relied on Masked Image Modeling (MIM). MIM models, while effective at reconstruction, suffer from a significant semantic gap, learning low-level textures rather than high-level concepts. Furthermore, they lack a language component, precluding zero-shot capabilities.
A major bottleneck is data scarcity; high-quality image-caption pairs for satellite imagery are rare compared to natural image datasets. Additionally, general-purpose models like the original CLIP struggle with the unique "aerial view" of satellite imagery, which provides an unobstructed, top-down perspective fundamentally different from ground-level photos.
Methods: Data Scaling via Annotation Unification
The core innovation of RemoteCLIP is the unification of heterogeneous annotations to scale pre-training data. Since natural language captions are scarce in RS, the authors developed two key strategies to "recycle" existing specialized datasets:
- Box-to-Caption (B2C): A rule-based system that converts object detection labels (bounding boxes and class names) into natural language descriptions based on object counts and spatial locations.
- Mask-to-Box (M2B): A pre-processing step that converts semantic segmentation masks into bounding boxes, which are then processed by the B2C pipeline.
By combining these converted datasets with UAV imagery and existing retrieval sets, the authors created a pre-training corpus 12 times larger than previous efforts. RemoteCLIP was then developed through continual pre-training on OpenAI’s original CLIP weights using the InfoNCE loss, aligning the model's existing language understanding with the specific visual domain of remote sensing.
Results and Discussion
RemoteCLIP achieves state-of-the-art (SOTA) performance across 16 downstream datasets. A key finding is that domain-specific alignment through continual pre-training is more impactful than raw model scale; for instance, a smaller RemoteCLIP (ResNet-50) can outperform a massive ViT-G model trained only on natural images.
In image-text retrieval, RemoteCLIP improved mean recall by 9.14% on RSITMD and 8.92% on RSICD. In zero-shot classification, it surpassed the vanilla CLIP baseline by an average of 6.39% accuracy. Furthermore, the newly introduced RemoteCount benchmark demonstrated that RemoteCLIP possesses a superior understanding of object quantities in aerial views compared to standard CLIP models.
Conclusion and Future Work
RemoteCLIP demonstrates that vision-language alignment is a powerful paradigm for remote sensing. While highly effective, limitations remain, such as performance on very low-resolution imagery and the relative simplicity of rule-based captions. Future work will explore the use of Large Language Models (LLMs) for more diverse caption generation, scaling to even larger vision transformers, and expanding the model's capabilities to multispectral and SAR data.
Summary updated