Open-Vocabulary Remote Sensing Image Semantic Segmentation

Qinglong Cao, Yuntian Chen, Chao Ma, Xiaokang YangarXiv 2024 · DOI · arXiv

Introduction

Traditional semantic segmentation models operate under a "closed-set" assumption, where the categories recognized during inference must strictly match those seen during training. This limitation is particularly restrictive in remote sensing (RS), where the diversity of geographic features and objects is vast. Open-Vocabulary Segmentation (OVS) offers a solution by leveraging vision-language models like CLIP to segment any category described via text prompts.

However, standard OVS frameworks designed for natural images struggle with the unique geometric properties of satellite and aerial imagery. Specifically, RS images exhibit significant orientation variance—objects like ships or airplanes can appear at any angle—and extreme scale variations, ranging from small vehicles to large industrial zones. This paper introduces the first OVS framework tailored specifically for these "earth vision" challenges.

Proposed Framework

The proposed architecture adapts the CLIP backbone to handle the spatial complexities of remote sensing data through three primary innovations: Rotation-Aggregative Similarity Computation (RSC), spatial/categorical refinement, and Scale-Aware Upsampling (SAU).

The proposed OVS framework for remote sensing, highlighting the Rotation-Aggregative Similarity Computation (RSC) and Scale-Aware Upsampling (SAU) modules.

Rotation-Aggregative Similarity Computation (RSC)

To address orientation invariance, the RSC module processes the input image at four canonical rotations ($0^\circ, 90^\circ, 180^\circ, 270^\circ$). Features extracted from these rotated views are compared against text embeddings to generate orientation-specific similarity maps. These maps are then rotated back to the original frame and aggregated. This ensures that the model's recognition capability is robust to the arbitrary heading of objects in overhead imagery.

Refinement and Scale-Aware Upsampling (SAU)

Since CLIP's global features are often too coarse for precise segmentation, the framework employs Swin Transformer layers for spatial and categorical refinement. To further improve detail, the Scale-Aware Upsampling (SAU) module integrates multi-level features from the CLIP vision encoder. By using semantic activation vectors, the model selectively emphasizes features at appropriate scales, allowing it to accurately delineate both fine-grained objects and large-scale land cover types.

Experimental Evaluation

The authors established a comprehensive OVS benchmark for remote sensing using four datasets: DLRSD, iSAID, Potsdam, and Vaihingen. The evaluation follows a cross-dataset protocol to test generalization: models are trained on one dataset and evaluated on the others using only category names as descriptions.

Quantitative Results

The proposed method demonstrates significant performance gains over existing state-of-the-art (SOTA) OVS models designed for natural images, such as CAT-Seg and SAN.

Cross-dataset evaluation results comparing the proposed method against SOTA OVS models across multiple remote sensing benchmarks.

As shown in the results table, when trained on DLRSD and tested on the complex iSAID dataset, the model achieves a 39.09% mIoU, representing a 15.53% improvement over CAT-Seg. Similar performance leads are observed across other metrics like Mean Accuracy (mACC) and Frequency Weighted IoU (fwIoU).

Key Insights

Ablation studies revealed that handling orientation (via RSC) provides a more substantial performance boost than handling scale in the RS context. This highlights that the "look" of an object relative to the sensor's grid is a primary bottleneck for zero-shot transfer in satellite imagery. Furthermore, the researchers found that four-axis rotation ($90^\circ$ increments) provides the optimal balance between feature richness and computational noise.

Conclusion

By explicitly modeling the geometric priors of remote sensing imagery, this work successfully bridges the gap between general-purpose vision-language models and specialized earth observation tasks. The proposed framework sets a new benchmark for open-vocabulary segmentation in RS, demonstrating that orientation-aware and scale-aware modules are essential for robust performance in bird's-eye view domains. Future work will focus on reducing the computational overhead of the multi-rotation inference and extending the model to non-RGB sensor data.

Summary updated

Abstract

Open-vocabulary image semantic segmentation (OVS) seeks to segment images into semantic regions across an open set of categories. Existing OVS methods commonly depend on foundational vision-language models and utilize similarity computation to tackle OVS tasks. However, these approaches are predominantly tailored to natural images and struggle with the unique characteristics of remote sensing images, such as rapidly changing orientations and significant scale variations. These challenges complicate OVS tasks in earth vision, requiring specialized approaches. To tackle this dilemma, we propose the first OVS framework specifically designed for remote sensing imagery, drawing inspiration from the distinct remote sensing traits. Particularly, to address the varying orientations, we introduce a rotation-aggregative similarity computation module that generates orientation-adaptive similarity maps as initial semantic maps. These maps are subsequently refined at both spatial and categorical levels to produce more accurate semantic maps. Additionally, to manage significant scale changes, we integrate multi-scale image features into the upsampling process, resulting in the final scale-aware semantic masks. To advance OVS in earth vision and encourage reproducible research, we establish the first open-sourced OVS benchmark for remote sensing imagery, including four public remote sensing datasets. Extensive experiments on this benchmark demonstrate our proposed method achieves state-of-the-art performance. All codes and datasets are available at https://github.com/caoql98/OVRS.
BibTeX
@misc{cao2024open,
  author = {Qinglong Cao and Yuntian Chen and Chao Ma and Xiaokang Yang},
  title = {{Open-Vocabulary Remote Sensing Image Semantic Segmentation}},
  year = {2024},
  eprint = {2409.07683},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.07683},
}
Figure