Insight Any Instance: Promptable Instance Segmentation for Remote Sensing Images

Xuexue Li, Wenhui Diao, Xinming Li, Xian SunIEEE TGRS 2025 · DOI

TL;DR

This paper introduces a novel "promptable" instance segmentation framework for remote sensing images (RSIs). By leveraging local and global "prompt" modules and a specialized loss function, it significantly improves segmentation accuracy, especially for small and imbalanced instances, while also enabling interactive segmentation with bounding box prompts.

What Problem Does This Paper Address?

Instance segmentation, the task of identifying and delineating each individual object in an image, is crucial for applications like land planning and intelligent transport. However, applying this to remote sensing images (RSIs) presents unique challenges:

  • Foreground-Background Imbalance: RSIs often have a very small proportion of foreground objects compared to the background, making it hard for models to learn effectively.
  • Limited Instance Size: Objects in RSIs can be very small, further complicating their detection and segmentation.
  • Harmful Downsampling: Traditional deep learning architectures for image analysis often use downsampling layers to reduce computational load. While this helps with spatial redundancy in natural images, it can discard crucial fine-grained details in RSIs, negatively impacting the segmentation of small or sparse objects.

Existing models struggle to overcome these issues, leading to limited performance. This paper aims to address these limitations by proposing a new approach that avoids harmful downsampling and specifically targets the unique characteristics of RSIs.

Methods

The core of the proposed solution is a novel prompt paradigm that enhances existing instance segmentation models. It consists of three main components:

  1. Local Prompt Module (LPM): This module mines local prompt information directly from the original, non-downsampled image. It focuses on the texture and structural details within the regions where instances are located, using a frequency-domain approach (FFT) to better exploit the information from different parts of an instance. This helps in understanding small instances without relying on downsampled features.
  2. Global-to-Local Prompt Module (GPM): This module models contextual information by bridging global image features with local instance regions. It uses a transformer-based architecture to extract global-to-local attention features, effectively enhancing the representation of specific instances by incorporating broader scene context without excessive computation. This is crucial for understanding instances that might lack sufficient individual texture.
  3. Proposal's Area Loss Function (PAreaLoss): To improve the quality of initial object proposals (which are used by LPM and GPM), a specialized loss function is introduced. It adds a "scale decoupling dimension" to the proposal regression, allowing the model to independently optimize for localization, scaling, and correction of proposals. This leads to more accurate proposals, which in turn benefits the prompt modules.

Crucially, this prompt paradigm can extend existing instance segmentation models to support promptable instance segmentation. This means users can provide bounding box prompts to guide the segmentation process, enabling interactive segmentation.

Datasets

The proposed method was evaluated on four benchmark datasets for remote sensing instance segmentation:

  • ISAID: A large-scale dataset of optical RSIs with 15 fine-grained categories.
  • NWPU VHR-10: An extension of an object detection dataset for instance segmentation, containing optical RSIs.
  • SSDD: A SAR image interpretation dataset primarily for ship detection and segmentation.
  • HRSID: A high-resolution SAR image dataset specifically for ship detection, segmentation, and instance segmentation tasks.

Key Insights and Ideas

  • Prompting for RSIs: The paper demonstrates that prompt learning, successful in natural image tasks, can be effectively adapted to address the unique challenges of RSIs, particularly foreground-background imbalance and small instance sizes.
  • Avoiding Downsampling is Key: The authors highlight that traditional downsampling in feature extraction is detrimental to RSI segmentation and propose methods that work with original image resolutions or mitigate its effects.
  • Global-Local Synergy: Combining local texture details with global context through dedicated prompt modules is essential for robust instance segmentation in RSIs.
  • Promptable Segmentation for RSIs: The framework successfully enables interactive instance segmentation using bounding box prompts, offering a practical advantage for users.

Key Results

  • Significant Accuracy Improvements: The proposed approach, when integrated with baseline models like Mask R-CNN, shows substantial improvements in both object detection (APd) and instance segmentation (APs) metrics across various datasets. For example, on ISAID, it achieved improvements of up to 4.7% in APs and 4.9% in APd.
  • Competitive Performance: The method achieves state-of-the-art or highly competitive results compared to existing methods on all tested datasets.
  • Efficiency: The promptable instance segmentation task can be performed very quickly, with a time consumption of only 40 ms per instance after feature extraction.
  • Ablation Studies: Experiments confirm the effectiveness of each proposed component (LPM, GPM, PAreaLoss), showing that their combination yields the best results.

Limitations and Future Work

  • Promptable Segmentation Improvement: While the promptable instance segmentation shows effectiveness, the improvement over automatic segmentation is modest in some cases. This is attributed to the fact that the automatic segmentation already misses fewer instances, but the promptable version still proves its utility.
  • Generalization and Versatility: The authors suggest future work could focus on improving the generalization and versatility of promptable remote sensing image instance segmentation models by developing more appropriate training strategies for large-scale datasets.

Summary updated

BibTeX
@article{li2025insight,
  author = {Xuexue Li and Wenhui Diao and Xinming Li and Xian Sun},
  title = {{Insight Any Instance: Promptable Instance Segmentation for Remote Sensing Images}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2025},
  doi = {10.1109/tgrs.2025.3543636},
}
Figure