Introduction
Traditional semantic segmentation models operate under a "closed-set" assumption, where the categories recognized during inference must strictly match those seen during training. This limitation is particularly restrictive in remote sensing (RS), where the diversity of geographic features and objects is vast. Open-Vocabulary Segmentation (OVS) offers a solution by leveraging vision-language models like CLIP to segment any category described via text prompts.
However, standard OVS frameworks designed for natural images struggle with the unique geometric properties of satellite and aerial imagery. Specifically, RS images exhibit significant orientation variance—objects like ships or airplanes can appear at any angle—and extreme scale variations, ranging from small vehicles to large industrial zones. This paper introduces the first OVS framework tailored specifically for these "earth vision" challenges.
Proposed Framework
The proposed architecture adapts the CLIP backbone to handle the spatial complexities of remote sensing data through three primary innovations: Rotation-Aggregative Similarity Computation (RSC), spatial/categorical refinement, and Scale-Aware Upsampling (SAU).
Rotation-Aggregative Similarity Computation (RSC)
To address orientation invariance, the RSC module processes the input image at four canonical rotations ($0^\circ, 90^\circ, 180^\circ, 270^\circ$). Features extracted from these rotated views are compared against text embeddings to generate orientation-specific similarity maps. These maps are then rotated back to the original frame and aggregated. This ensures that the model's recognition capability is robust to the arbitrary heading of objects in overhead imagery.
Refinement and Scale-Aware Upsampling (SAU)
Since CLIP's global features are often too coarse for precise segmentation, the framework employs Swin Transformer layers for spatial and categorical refinement. To further improve detail, the Scale-Aware Upsampling (SAU) module integrates multi-level features from the CLIP vision encoder. By using semantic activation vectors, the model selectively emphasizes features at appropriate scales, allowing it to accurately delineate both fine-grained objects and large-scale land cover types.
Experimental Evaluation
The authors established a comprehensive OVS benchmark for remote sensing using four datasets: DLRSD, iSAID, Potsdam, and Vaihingen. The evaluation follows a cross-dataset protocol to test generalization: models are trained on one dataset and evaluated on the others using only category names as descriptions.
Quantitative Results
The proposed method demonstrates significant performance gains over existing state-of-the-art (SOTA) OVS models designed for natural images, such as CAT-Seg and SAN.
As shown in the results table, when trained on DLRSD and tested on the complex iSAID dataset, the model achieves a 39.09% mIoU, representing a 15.53% improvement over CAT-Seg. Similar performance leads are observed across other metrics like Mean Accuracy (mACC) and Frequency Weighted IoU (fwIoU).
Key Insights
Ablation studies revealed that handling orientation (via RSC) provides a more substantial performance boost than handling scale in the RS context. This highlights that the "look" of an object relative to the sensor's grid is a primary bottleneck for zero-shot transfer in satellite imagery. Furthermore, the researchers found that four-axis rotation ($90^\circ$ increments) provides the optimal balance between feature richness and computational noise.
Conclusion
By explicitly modeling the geometric priors of remote sensing imagery, this work successfully bridges the gap between general-purpose vision-language models and specialized earth observation tasks. The proposed framework sets a new benchmark for open-vocabulary segmentation in RS, demonstrating that orientation-aware and scale-aware modules are essential for robust performance in bird's-eye view domains. Future work will focus on reducing the computational overhead of the multi-rotation inference and extending the model to non-RGB sensor data.
Summary updated