A Large-Scale Referring Remote Sensing Image Segmentation Dataset and Benchmark

Zhigang Yang, Huiguang Yao, Linmao Tian, Xuezhi Zhao, Qiang Li, Qi Wang

TL;DR

Researchers have released NWPU-Refer, the largest and most diverse dataset for Referring Remote Sensing Image Segmentation (RRSIS), featuring over 15,000 high-resolution images with manual bilingual annotations. Alongside the dataset, they introduced MRSNet, a multi-scale network that leverages spatial-spectral refinement and hierarchical feature fusion to significantly outperform existing models in identifying specific objects from natural language descriptions.

What Problem Does This Paper Address?

Referring Remote Sensing Image Segmentation (RRSIS) is the task of taking a natural language query (e.g., "the small white boat docked near the green pier") and generating a precise pixel-level mask of that object in a satellite or aerial image.

This is exceptionally difficult for two reasons:

  1. Remote Sensing Complexity: Unlike standard photos, satellite imagery involves massive scale variations (a car vs. an airport), cluttered backgrounds, and diverse geographic contexts.
  2. Dataset Limitations: Existing RRSIS datasets are often small, low-resolution, or rely on "noisy" automated labels from models like SAM (Segment Anything Model). They also typically focus on simple scenes with a single, obvious target, failing to represent the "multi-target" or "no-target" (negative) scenarios found in the real world.

Methods

The authors propose a new framework called the Multi-scale Referring Segmentation Network (MRSNet). It uses a Swin-Transformer for visual features and BERT for linguistic features, but its "secret sauce" lies in two novel modules:

  • Intra-scale Feature Interaction Module (IFIM): Integrated into each encoder stage, this module performs Pyramidal Spatial-Spectral Refinement. It doesn't just look at pixels; it uses Fourier Transforms to analyze the "frequency domain" of the image, helping the model distinguish textures and patterns that look similar in standard spatial views. It also includes a Context-Aware Spatial Relation submodule to model how different objects in a scene relate to one another.
  • Hierarchical Feature Interaction Module (HFIM): This handles the "multi-scale" problem. It uses PixelShuffle (a technique to trade channel depth for spatial resolution) and combined spatial/frequency self-attention to fuse features across different layers of the network without losing fine-grained spatial details.

Datasets

The paper introduces NWPU-Refer, which sets a new bar for the field:

  • Scale: 15,003 images and 49,745 annotated targets.
  • Resolution: High-res (1024–2048px), far exceeding previous benchmarks.
  • Diversity: Images from 30+ countries across five continents, covering 32 categories (from "wind turbines" to "swimming pools").
  • Complexity: Includes single-object, multi-object, and "no-target" scenarios (where the model must correctly produce an empty mask if the described object isn't present).
  • Bilingual: Annotations are provided in both English and Chinese.

Key Insights and Ideas

  • The Spectral Advantage: One of the most interesting "aha" moments is the use of frequency-domain analysis. By looking at the spectral components of an image, the model can better separate man-made objects from natural backgrounds that might have similar colors but different periodic textures.
  • Manual Labeling Matters: While many modern datasets use automated tools like SAM to scale up, the authors argue that manual annotation is still essential for remote sensing. Automated tools often miss fine-grained boundaries or small objects (like power line towers) that are critical for high-stakes applications.
  • Negative Reasoning: By including "no-target" queries, the authors force the model to move beyond "saliency detection" (just finding the most interesting thing in the box) and toward true linguistic comprehension.

Key Results

MRSNet was compared against current state-of-the-art models LAVT and RMSIN on the NWPU-Refer benchmark.

  • Quantitative: MRSNet achieved a mean IoU (mIoU) of 43.60% and an overall IoU (oIoU) of 57.04% on the test set.
  • Precision: At a strict threshold ([email protected]), MRSNet outperformed the previous best model (RMSIN) by a notable margin, showing it is much better at capturing the exact shape of the target.
  • Qualitative: Visual results show MRSNet successfully distinguishing between "all buildings" and "all airplanes" in complex airport scenes where other models often confused the two.

Limitations and Future Work

While a significant step forward, the paper notes a few areas for growth:

  • Precision Ceiling: Even with these improvements, the precision at high IoU thresholds ([email protected]) remains low (~8.5%), indicating that "pixel-perfect" segmentation of tiny objects from space is still an unsolved problem.
  • Computational Cost: The use of Transformers and dual-domain (spatial and spectral) processing is computationally intensive, which might limit real-time deployment on edge devices or satellites.
  • Language Expansion: While bilingual, the dataset could be expanded to more languages to support truly global, localized disaster response or urban planning tools.

Summary updated

Abstract

Referring Remote Sensing Image Segmentation is a complex and challenging task that integrates the paradigms of computer vision and natural language processing. Existing datasets for RRSIS suffer from critical limitations in resolution, scene diversity, and category coverage, which hinders the generalization and real-world applicability of refer segmentation models. To facilitate the development of this field, we introduce NWPU-Refer, the largest and most diverse RRSIS dataset to date, comprising 15,003 high-resolution images (1024-2048px) spanning 30+ countries with 49,745 annotated targets supporting single-object, multi-object, and non-object segmentation scenarios. Additionally, we propose the Multi-scale Referring Segmentation Network (MRSNet), a novel framework tailored for the unique demands of RRSIS. MRSNet introduces two key innovations: (1) an Intra-scale Feature Interaction Module (IFIM) that captures fine-grained details within each encoder stage, and (2) a Hierarchical Feature Interaction Module (HFIM) to enable seamless cross-scale feature fusion, preserving spatial integrity while enhancing discriminative power. Extensive experiments conducte on the proposed NWPU-Refer dataset demonstrate that MRSNet achieves state-of-the-art performance across multiple evaluation metrics, validating its effectiveness. The dataset and code are publicly available at https://github.com/CVer-Yang/NWPU-Refer.
BibTeX
@misc{yang2025large,
  author = {Zhigang Yang and Huiguang Yao and Linmao Tian and Xuezhi Zhao and Qiang Li and Qi Wang},
  title = {{A Large-Scale Referring Remote Sensing Image Segmentation Dataset and Benchmark}},
  year = {2025},
  eprint = {2506.03583},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.03583},
}
Figure