SCORE: Scene Context Matters in Open-Vocabulary Remote Sensing Instance Segmentation

Shiqi Huang, Shuting He, Huaiyuan Qin, Bihan Wen

Abstract

Most existing remote sensing instance segmentation approaches are designed for close-vocabulary prediction, limiting their ability to recognize novel categories or generalize across datasets. This restricts their applicability in diverse Earth observation scenarios. To address this, we introduce open-vocabulary (OV) learning for remote sensing instance segmentation. While current OV segmentation models perform well on natural image datasets, their direct application to remote sensing faces challenges such as diverse landscapes, seasonal variations, and the presence of small or ambiguous objects in aerial imagery. To overcome these challenges, we propose SCORE (Scene Context matters in Open-vocabulary REmote sensing instance segmentation), a framework that integrates multi-granularity scene context, i.e., regional context and global context, to enhance both visual and textual representations. Specifically, we introduce Region-Aware Integration, which refines class embeddings with regional context to improve object distinguishability. Additionally, we propose Global Context Adaptation, which enriches naive text embeddings with remote sensing global context, creating a more adaptable and expressive linguistic latent space for the classifier. We establish new benchmarks for OV remote sensing instance segmentation across diverse datasets. Experimental results demonstrate that our proposed method achieves SOTA performance, providing a robust solution for large-scale, real-world geospatial analysis.

Introduction

Remote sensing (RS) instance segmentation is traditionally limited by a "closed-vocabulary" approach, where models can only identify categories seen during training. While Open-Vocabulary (OV) models like CLIP offer a solution by leveraging vision-language pre-training, they often struggle with the unique characteristics of aerial imagery. In satellite views, objects are frequently small, densely packed, and visually ambiguous. For instance, a white rectangular shape could be either a vehicle or a ship depending entirely on its environment.

The importance of surrounding environment in identifying ambiguous objects in remote sensing.

Methodology: The SCORE Framework

To address these challenges, we propose SCORE. The framework enhances standard OV segmentation by injecting multi-granularity scene context into the model's decision-making process through two primary components.

Overview of the SCORE architecture, featuring Region-Aware Integration (RAI) and Global Context Adaptation (GCA).

Region-Aware Integration (RAI)

The RAI module refines object-level features by incorporating regional context. It utilizes Adaptive Region Formation, which employs a learnable dilation factor to expand predicted masks. This allows the model to "look around" an object—capturing the water around a ship or the asphalt around a car—to refine its visual embedding and improve class distinguishability.

Global Context Adaptation (GCA)

The GCA module focuses on the linguistic side of the problem. It enriches standard CLIP text embeddings with global scene features extracted via a remote-sensing-specific CLIP model (RemoteCLIP). By injecting visual global context into the text latent space, SCORE helps the classifier understand that a "vehicle" in a desert environment looks different from one in a snowy urban setting, making the model more adaptable to environmental variations.

Key Results and Insights

We established new benchmarks for OV remote sensing instance segmentation using datasets like iSAID and SIOR. Our framework consistently achieved state-of-the-art (SOTA) performance:

  • Performance Boost: SCORE improved mean Average Precision (mAP) by an average of 5.53% on iSAID and 4.32% on SIOR compared to previous leaders like FC-CLIP.
  • Generalization: In cross-dataset evaluations (e.g., training on SIOR and testing on iSAID), SCORE demonstrated superior robustness, outperforming baselines by up to 12.55%.
  • Novel Class Recognition: The model successfully identified and segmented categories not present in its training set, such as football fields, chimneys, and overpasses, effectively bridging the domain gap between training data and real-world diversity.

Conclusion

SCORE demonstrates that scene context is a critical missing piece in open-vocabulary remote sensing. By integrating regional and global context, the framework provides a robust solution for large-scale geospatial analysis, enabling the identification of novel objects in complex, varied environments.

Summary updated

Abstract

Most existing remote sensing instance segmentation approaches are designed for close-vocabulary prediction, limiting their ability to recognize novel categories or generalize across datasets. This restricts their applicability in diverse Earth observation scenarios. To address this, we introduce open-vocabulary (OV) learning for remote sensing instance segmentation. While current OV segmentation models perform well on natural image datasets, their direct application to remote sensing faces challenges such as diverse landscapes, seasonal variations, and the presence of small or ambiguous objects in aerial imagery. To overcome these challenges, we propose $\textbf{SCORE}$ ($\textbf{S}$cene $\textbf{C}$ontext matters in $\textbf{O}$pen-vocabulary $\textbf{RE}$mote sensing instance segmentation), a framework that integrates multi-granularity scene context, i.e., regional context and global context, to enhance both visual and textual representations. Specifically, we introduce Region-Aware Integration, which refines class embeddings with regional context to improve object distinguishability. Additionally, we propose Global Context Adaptation, which enriches naive text embeddings with remote sensing global context, creating a more adaptable and expressive linguistic latent space for the classifier. We establish new benchmarks for OV remote sensing instance segmentation across diverse datasets. Experimental results demonstrate that, our proposed method achieves SOTA performance, which provides a robust solution for large-scale, real-world geospatial analysis. Our code is available at https://github.com/HuangShiqi128/SCORE.
BibTeX
@inproceedings{huang2025score,
  author = {Shiqi Huang and Shuting He and Huaiyuan Qin and Bihan Wen},
  title = {{SCORE: Scene Context Matters in Open-Vocabulary Remote Sensing Instance Segmentation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}
Figure