Abstract
Referring Remote Sensing Image Segmentation (RRSIS) is a new challenge that combines computer vision and natural language processing, delineating specific regions in aerial images as described by textual queries. Traditional Referring Image Segmentation (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery, leading to suboptimal segmentation results. To address these challenges, we introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS. RMSIN incorporates an Intra-scale Interaction Module (IIM) to effectively address the fine-grained detail required at multiple scales and a Cross-scale Interaction Module (CIM) for integrating these details coherently across the network. Furthermore, RMSIN employs an Adaptive Rotated Convolution (ARC) to account for the diverse orientations of objects, a novel contribution that significantly enhances segmentation accuracy. To assess the efficacy of RMSIN, we have curated an expansive dataset comprising 17,402 image-caption-mask triplets, which is unparalleled in terms of scale and variety. This dataset not only presents the model with a wide range of spatial and rotational scenarios but also establishes a stringent benchmark for the RRSIS task, ensuring a rigorous evaluation of performance. Our experimental evaluations demonstrate the exceptional performance of RMSIN, surpassing existing state-of-the-art models by a significant margin.
Introduction
In the field of computer vision, Referring Image Segmentation (RIS) is the art of teaching a model to find an object based on a text prompt—like "the red car parked near the tree." While significant strides have been made in natural images (such as street views or everyday photography), applying this to satellite and aerial imagery is a fundamentally different challenge.
A new approach from researchers at Xiamen University introduces the Rotated Multi-Scale Interaction Network (RMSIN), a framework specifically designed to handle the unique "top-down" challenges of remote sensing. Unlike standard imagery, remote sensing (RS) data is characterized by extreme scale variations and arbitrary object orientations that traditional models are not equipped to handle.
The Challenges of Remote Sensing
Traditional RIS models struggle with remote sensing for two primary reasons:
- Extreme Scale Variation: In a single satellite pass, a model might need to segment a tiny vehicle and a massive stadium simultaneously. Standard models often lose the fine-grained details of small objects or the global context necessary to identify large ones.
- Arbitrary Orientation: Unlike photos taken on the ground where "up" is usually up, objects in aerial images can be oriented in any direction. A standard horizontal convolution kernel is not optimized for a boat docked at a 45-degree angle or a plane parked diagonally on a runway.
Furthermore, the field has been hampered by a lack of data. Previous datasets were relatively small (around 4,000 images), making it difficult to train robust, data-hungry transformer models.
The RMSIN Framework
The authors propose the RMSIN framework, which features three core innovations to bridge the gap between natural language and complex aerial imagery:
- Intra-scale Interaction Module (IIM): This module works within individual layers of the network to refine features. It utilizes a "Various Receptive Branch" to capture different levels of detail and a "Cross-modal Alignment Branch" to ensure visual features are accurately aligned with the text description.
- Cross-scale Interaction Module (CIM): This acts as a bridge between different layers. It ensures that high-level semantic information (e.g., "this is a track field") and low-level structural details (e.g., "here is the white boundary line") are integrated coherently using a "Scale-aware Gate."
- Adaptive Rotated Convolution (ARC): This is the standout feature. Instead of using fixed, square kernels, ARC predicts the orientation of objects and dynamically "rotates" the convolution weights. This allows the model to align its filters with the physical orientation of the objects in the image, leading to much sharper boundary definitions.
A New Benchmark: RRSIS-D
The researchers also introduced RRSIS-D, a dataset that is currently unparalleled in the field. It contains 17,402 image-caption-mask triplets, nearly four times larger than previous standards. The dataset includes 20 categories and 7 attributes, with a standardized 800x800 resolution. To ensure high fidelity, the team leveraged the Segment Anything Model (SAM) to semi-automate mask generation, followed by meticulous manual refinement.
Key Results and Insights
RMSIN sets a new benchmark across the board. On the RRSIS-D test set, it achieved a 64.20% mIoU (Mean Intersection-over-Union), outperforming the previous leader, LAVT, by over 3%. The gains were even more dramatic at higher precision thresholds, such as [email protected], where RMSIN outperformed LAVT by nearly 2.6%.
Qualitative visualizations show that while previous models often miss parts of rotated objects or produce blurry masks for tiny objects, RMSIN produces crisp, well-aligned segmentations even in noisy backgrounds. This suggests that rotation should be treated as a first-class citizen in model architecture rather than just a data augmentation technique.
Conclusion
By building rotation and scale awareness directly into the network architecture, RMSIN provides a robust solution for the unique demands of remote sensing. The combination of the specialized ARC module and the expansive RRSIS-D dataset establishes a new state-of-the-art for the field, opening doors for more complex aerial tracking and change detection tasks in the future.
Summary updated