Abstract
Most existing remote sensing instance segmentation approaches are designed for close-vocabulary prediction, limiting their ability to recognize novel categories or generalize across datasets. This restricts their applicability in diverse Earth observation scenarios. To address this, we introduce open-vocabulary (OV) learning for remote sensing instance segmentation. While current OV segmentation models perform well on natural image datasets, their direct application to remote sensing faces challenges such as diverse landscapes, seasonal variations, and the presence of small or ambiguous objects in aerial imagery. To overcome these challenges, we propose SCORE (Scene Context matters in Open-vocabulary REmote sensing instance segmentation), a framework that integrates multi-granularity scene context, i.e., regional context and global context, to enhance both visual and textual representations. Specifically, we introduce Region-Aware Integration, which refines class embeddings with regional context to improve object distinguishability. Additionally, we propose Global Context Adaptation, which enriches naive text embeddings with remote sensing global context, creating a more adaptable and expressive linguistic latent space for the classifier. We establish new benchmarks for OV remote sensing instance segmentation across diverse datasets. Experimental results demonstrate that our proposed method achieves SOTA performance, providing a robust solution for large-scale, real-world geospatial analysis.
Introduction
Remote sensing (RS) instance segmentation is traditionally limited by a "closed-vocabulary" approach, where models can only identify categories seen during training. While Open-Vocabulary (OV) models like CLIP offer a solution by leveraging vision-language pre-training, they often struggle with the unique characteristics of aerial imagery. In satellite views, objects are frequently small, densely packed, and visually ambiguous. For instance, a white rectangular shape could be either a vehicle or a ship depending entirely on its environment.
Methodology: The SCORE Framework
To address these challenges, we propose SCORE. The framework enhances standard OV segmentation by injecting multi-granularity scene context into the model's decision-making process through two primary components.
Region-Aware Integration (RAI)
The RAI module refines object-level features by incorporating regional context. It utilizes Adaptive Region Formation, which employs a learnable dilation factor to expand predicted masks. This allows the model to "look around" an object—capturing the water around a ship or the asphalt around a car—to refine its visual embedding and improve class distinguishability.
Global Context Adaptation (GCA)
The GCA module focuses on the linguistic side of the problem. It enriches standard CLIP text embeddings with global scene features extracted via a remote-sensing-specific CLIP model (RemoteCLIP). By injecting visual global context into the text latent space, SCORE helps the classifier understand that a "vehicle" in a desert environment looks different from one in a snowy urban setting, making the model more adaptable to environmental variations.
Key Results and Insights
We established new benchmarks for OV remote sensing instance segmentation using datasets like iSAID and SIOR. Our framework consistently achieved state-of-the-art (SOTA) performance:
- Performance Boost: SCORE improved mean Average Precision (mAP) by an average of 5.53% on iSAID and 4.32% on SIOR compared to previous leaders like FC-CLIP.
- Generalization: In cross-dataset evaluations (e.g., training on SIOR and testing on iSAID), SCORE demonstrated superior robustness, outperforming baselines by up to 12.55%.
- Novel Class Recognition: The model successfully identified and segmented categories not present in its training set, such as football fields, chimneys, and overpasses, effectively bridging the domain gap between training data and real-world diversity.
Conclusion
SCORE demonstrates that scene context is a critical missing piece in open-vocabulary remote sensing. By integrating regional and global context, the framework provides a robust solution for large-scale geospatial analysis, enabling the identification of novel objects in complex, varied environments.
Summary updated