RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Models

Keyan Chen, Chenyang Liu, Bowen Chen, Jiafan Zhang, Zhengxia Zou, Zhenwei ShiarXiv 2025 · DOI · arXiv

TL;DR

RSRefSeg 2 revolutionizes Referring Remote Sensing Image Segmentation (RRSIS) by decoupling the task into a two-stage pipeline: coarse localization followed by fine segmentation. By leveraging the combined strengths of CLIP for language-vision alignment and SAM for zero-shot segmentation, alongside a novel cascaded prompter to resolve textual ambiguity, it achieves state-of-the-art performance across major remote sensing benchmarks.

What Problem Does This Paper Address?

Imagine asking a satellite AI to "find the small white vehicle parked to the right of the large gray airplane." This task—known as Referring Remote Sensing Image Segmentation (RRSIS)—requires a model to understand natural language and pinpoint exact pixel boundaries in overhead imagery.

Traditional approaches try to solve this using a unified, three-stage pipeline that handles feature encoding, cross-modal interaction, and pixel decoding all at once. The authors argue that this architectural coupling conflates two fundamentally different subtasks: figuring out where the target is (localization) and determining how to outline its boundaries (segmentation).

When a text description is complex or ambiguous, traditional models suffer from severe error propagation. For instance, if a native vision-language model like CLIP encounters multiple similar objects, it often activates incorrect or multiple regions, confusing the pixel decoder and ruining the final mask.

Fig. 1: (a) Conventional: three-stage (dual encoders, cross- interaction, pixel decoder). (b) RSRefSeg 2: collaborative coarse localization and fine segmentation.

Methods

To fix this, RSRefSeg 2 introduces a collaborative dual-stage framework: "coarse localization followed by fine segmentation."

  1. Dual-Modality Aligned Encoder: The model uses a pre-trained CLIP model to map low-resolution images and text descriptions into a shared semantic space, establishing a rough idea of where the target lies.
  2. Cascaded Second-Order Prompter: To prevent CLIP from misactivating multiple objects in complex scenes, the authors designed a prompter that breaks down the text embeddings into distinct, complementary semantic subspaces. Through implicit cascaded reasoning, the first subspace identifies candidate regions, and the second refines them based on visual features. This generates precise, SAM-compatible sparse (point/box) and dense (mask) prompts.
  3. Prompt-Guided Refinement Mask Generator: These prompts are fed into the Segment Anything Model (SAM), which processes the original high-resolution image to output a sharp, pixel-level binary mask.

To bridge the domain gap between natural images (which CLIP and SAM were trained on) and remote sensing data, the authors utilize Parameter-Efficient Fine-Tuning (PEFT) via Low-Rank Adaptation (LoRA), keeping the bulk of the foundation models frozen while tuning lightweight adapter modules.

Detailed architecture of RSRefSeg 2, highlighting the cascaded prompter and SAM-based mask generation.

Datasets

The framework was rigorously evaluated on three primary RRSIS datasets:

  • RefSegRS: Contains 4,420 image-description-mask triplets across 14 hierarchical categories with a high spatial resolution of 0.13 meters/pixel.
  • RRSIS-D: A larger dataset featuring 17,402 triplets and 20 categories, characterized by massive scale variations in target objects.
  • RISBench: The largest benchmark used, containing 52,472 triplets across 26 categories, presenting significant lexical diversity and syntactic complexity.

Key Insights and Ideas

  • The Power of Decoupling: Separating "where" from "how" isolates textual comprehension errors from visual delineation errors. This makes the model highly interpretable and easier to debug.
  • Subspace Decomposition for Reasoning: Instead of forcing a model to understand a complex sentence in one shot, decomposing text embeddings into orthogonal subspaces allows the model to perform implicit step-by-step reasoning (e.g., locating an airplane first, then finding the vehicle next to it).
  • Asymmetric Domain Shift: The ablation studies revealed that vision encoders (CLIP-v and SAM-v) require higher LoRA ranks than text encoders. This is because text descriptions of remote sensing scenes don't differ fundamentally from natural language, but overhead satellite imagery looks vastly different from standard internet photos.

Key Results

RSRefSeg 2 achieved dominant performance across all three datasets, outperforming both traditional CNN-Transformer hybrids and contemporary foundation-model-based architectures.

  • RefSegRS: Achieved 81.24% cIoU (cumulative Intersection over Union) and 77.39% gIoU (generalized IoU), representing a substantial ~3% gIoU improvement over the previous state-of-the-art.
  • RRSIS-D & RISBench: Secured top marks across nearly all precision thresholds, demonstrating exceptional robustness against extreme scale variations and small-object detection (like windmills and small vehicles).
TABLE II: Performance comparison across various evaluation metrics on the RefSegRS test dataset.

Limitations and Future Work

Despite its success, RSRefSeg 2 hits a wall when descriptions involve highly compound logic or nested spatial relationships. Furthermore, because SAM relies heavily on distinct object boundaries learned from natural images, RSRefSeg 2 can struggle in low-contrast remote sensing scenarios or on targets with ambiguous boundaries, such as harbors and dams.

The authors suggest two future directions: integrating reinforcement learning frameworks to boost the model's explicit semantic reasoning, and developing alternative segmentation decoders specifically engineered to handle the unique boundary fuzziness of geospatial imagery.

Summary updated

Abstract

Referring Remote Sensing Image Segmentation provides a flexible and fine-grained framework for remote sensing scene analysis via vision-language collaborative interpretation. Current approaches predominantly utilize a three-stage pipeline encompassing dual-modal encoding, cross-modal interaction, and pixel decoding. These methods demonstrate significant limitations in managing complex semantic relationships and achieving precise cross-modal alignment, largely due to their coupled processing mechanism that conflates target localization with boundary delineation. This architectural coupling amplifies error propagation under semantic ambiguity while restricting model generalizability and interpretability. To address these issues, we propose RSRefSeg 2, a decoupling paradigm that reformulates the conventional workflow into a collaborative dual-stage framework: coarse localization followed by fine segmentation. RSRefSeg 2 integrates CLIP's cross-modal alignment strength with SAM's segmentation generalizability through strategic foundation model collaboration. Specifically, CLIP is employed as the dual-modal encoder to activate target features within its pre-aligned semantic space and generate localization prompts. To mitigate CLIP's misactivation challenges in multi-entity scenarios described by referring texts, a cascaded second-order prompter is devised, which enhances precision through implicit reasoning via decomposition of text embeddings into complementary semantic subspaces. These optimized semantic prompts subsequently direct the SAM to generate pixel-level refined masks, thereby completing the semantic transmission pipeline. Extensive experiments (RefSegRS, RRSIS-D, and RISBench) demonstrate that RSRefSeg 2 surpasses contemporary methods in segmentation accuracy (+~3% gIoU) and complex semantic interpretation. Code is available at: https://github.com/KyanChen/RSRefSeg2.
BibTeX
@misc{chen2025rsrefseg,
  author = {Keyan Chen and Chenyang Liu and Bowen Chen and Jiafan Zhang and Zhengxia Zou and Zhenwei Shi},
  title = {{RSRefSeg 2: Decoupling Referring Remote Sensing Image Segmentation with Foundation Models}},
  year = {2025},
  eprint = {2507.06231},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2507.06231},
}

Loading related content...

Figure