RSRefSeg: Referring Remote Sensing Image Segmentation with Foundation Models

Keyan Chen, Jiafan Zhang, Chenyang Liu, Zhengxia Zou, Zhenwei ShiarXiv 2025 · arXiv

TL;DR

RSRefSeg is a novel foundational framework for referring remote sensing image segmentation that successfully bridges the semantic power of CLIP with the precise masking capabilities of the Segment Anything Model (SAM). By introducing a specialized "AttnPrompter" module and utilizing parameter-efficient fine-tuning, the model accurately isolates and segments specific objects in overhead imagery based on free-form natural language descriptions, achieving state-of-the-art performance.

What Problem Does This Paper Address?

Traditional remote sensing image segmentation relies on predefined, rigid object classes (e.g., "building," "road"). Referring remote sensing image segmentation (RRSIS) steps beyond this limitation by allowing users to locate and segment objects using flexible, natural language queries—such as "the gray and black vehicle parked next to the stadium."

However, current RRSIS methods face two major hurdles:

  1. Fine-Grained Alignment: It is incredibly difficult to map abstract textual concepts to multi-scale, often tiny objects captured from satellite or aerial perspectives.
  2. Domain Shift: Powerful vision-language foundation models like CLIP and SAM are trained on everyday, ground-level images. When applied directly to top-down remote sensing data, their performance drops significantly.

The authors introduce RSRefSeg to solve these issues, effectively transferring general-domain foundation knowledge to complex remote sensing tasks.

Methods

RSRefSeg combines the strengths of two foundational models through a decoupled architecture consisting of three main components:

  1. Fine-tuned CLIP (SigLIP): The authors use an enhanced version of CLIP (specifically siglip-so400m-patch14-384) to extract both global and local text semantics alongside low-resolution visual features. Crucially, they remove CLIP's final pooling layers to preserve spatial feature maps.
  2. AttnPrompter: This novel module acts as a bridge. It uses the text embeddings as "filters" to highlight referring-related visual features in the latent space, generating a Referring Activated Map. It then processes this map into two types of prompts compatible with SAM: sparse prompts (representing point or box coordinates) and dense prompts (coarse segmentation masks).
  3. Fine-tuned SAM: SAM takes the original high-resolution image and the sparse/dense prompts generated by the AttnPrompter to output a highly refined, pixel-perfect binary segmentation mask.
Fig. 1. The overview of the proposed RSRefSeg. The fire icon symbolizes that the model parameters are tuned.

To combat domain shift without destroying the models' pre-trained knowledge, the authors integrate Low-Rank Adaptation (LoRA) fine-tuning ($r=16$) into both the CLIP and SAM encoders, keeping the bulk of the 1.2-billion-parameter model frozen.

Datasets

The model was trained and evaluated on the RRSIS-D dataset, a benchmark designed specifically for referring remote sensing image segmentation.

  • Size: 17,402 triplets (image, mask, and natural language description), split into 12,181 training, 1,740 validation, and 3,481 testing samples.
  • Diversity: Includes 20 distinct semantic categories, such as aircraft, stadiums, oil tanks, and highway service areas, with spatial resolutions varying widely from 0.5 to 30 meters.

Key Insights and Ideas

  • Decoupled Perception and Segmentation: Instead of forcing a single network to master both language comprehension and pixel-perfect edge detection, RSRefSeg delegates coarse semantic targeting to CLIP and fine-grained boundary delineation to SAM.
  • Text Encoders Benefit from Tuning: Conventional wisdom often assumes that language models don't need domain-specific tuning because text is domain-invariant. However, the authors found that fine-tuning CLIP's text encoder significantly boosted performance. This is because remote sensing queries are typically short, attribute-heavy phrases rather than the full sentences CLIP was originally trained on.
  • Prompt Balance is Critical: SAM's performance degrades if its input prompts deviate too much from its original training distribution. The authors discovered that an intermediate spatial downsampling rate in the prompter yields the best results—too many or too few prompt points actually hurt accuracy.

Key Results

RSRefSeg was evaluated against several state-of-the-art (SOTA) models using generalized Intersection over Union (gIoU) and cumulative Intersection over Union (cIoU).

Quantitative comparison with state-of-the-art methods on the RRSIS-D dataset.
  • SOTA Performance: The large variant, RSRefSeg-l, achieved a cIoU of 77.24 and a gIoU of 64.67, outperforming previous remote-sensing-specific models like FIANet and RMISN.
  • High-Precision Edge: At the strictest evaluation metric ([email protected], requiring a 90% overlap with the ground truth), RSRefSeg-l achieved a score of 30.80, a substantial leap over FIANet's 24.13. This proves the framework's exceptional ability to generate highly accurate boundaries for complex overhead targets.

Limitations and Future Work

While RSRefSeg delivers impressive results, the paper notes a few areas for refinement:

  • Prompt Sensitivity: The model relies heavily on the AttnPrompter finding a "sweet spot" for prompt density; deviations can cause SAM to output suboptimal masks.
  • Computational Footprint: Operating two massive foundational backbones (CLIP + SAM) simultaneously introduces significant parameter weight (1.2 billion parameters). Although the authors do not deeply analyze inference latency, the computational overhead could limit deployment in real-time or resource-constrained edge applications. Future work could explore distilling this framework into a more compact, single-backbone model.

Summary updated

Abstract

Referring remote sensing image segmentation is crucial for achieving fine-grained visual understanding through free-format textual input, enabling enhanced scene and object extraction in remote sensing applications. Current research primarily utilizes pre-trained language models to encode textual descriptions and align them with visual modalities, thereby facilitating the expression of relevant visual features. However, these approaches often struggle to establish robust alignments between fine-grained semantic concepts, leading to inconsistent representations across textual and visual information. To address these limitations, we introduce a referring remote sensing image segmentation foundational model, RSRefSeg. RSRefSeg leverages CLIP for visual and textual encoding, employing both global and local textual semantics as filters to generate referring-related visual activation features in the latent space. These activated features then serve as input prompts for SAM, which refines the segmentation masks through its robust visual generalization capabilities. Experimental results on the RRSIS-D dataset demonstrate that RSRefSeg outperforms existing methods, underscoring the effectiveness of foundational models in enhancing multimodal task comprehension. The code is available at \url{https://github.com/KyanChen/RSRefSeg}.
BibTeX
@misc{chen2025rsrefseg,
  author = {Keyan Chen and Jiafan Zhang and Chenyang Liu and Zhengxia Zou and Zhenwei Shi},
  title = {{RSRefSeg: Referring Remote Sensing Image Segmentation with Foundation Models}},
  year = {2025},
  eprint = {2501.06809},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.06809},
}

Loading related content...

Figure