Abstract
The Segment Anything Model (SAM) has set a new standard for interactive segmentation in natural images but faces significant challenges in the remote sensing domain. This paper introduces ROS-SAM, a specialized adaptation designed to segment moving objects in satellite video with high precision. By employing Low-Rank Adaptation (LoRA), an optimized high-quality mask decoder, and a "crop-and-upsample" inference strategy, ROS-SAM achieves a 13% IoU improvement over the base SAM model and demonstrates impressive zero-shot performance on unseen tracking datasets.
Introduction
Interactive segmentation in remote sensing satellite video (RSVMO) is critical for applications ranging from traffic monitoring to maritime surveillance. However, the original SAM model struggles with three primary issues:
- Domain Gap: Satellite imagery lacks the standard orientation found in natural images, and objects often have ambiguous features.
- Scale Issues: Moving objects like cars and planes are often extremely small, leading to information loss during SAM's standard 1024x1024 resizing.
- Mask Quality: The default decoder often produces fragmented or coarse boundaries, which is insufficient for high-precision remote sensing tasks.
ROS-SAM addresses these hurdles to enable the efficient creation of high-quality pixel-level datasets from existing tracking data with minimal manual effort.
Methodology
The architecture of ROS-SAM incorporates three key innovations to transform the base SAM model into a robust remote sensing tool:
LoRA-based Encoder Tuning
To bridge the domain gap without the computational cost of full fine-tuning, ROS-SAM utilizes Low-Rank Adaptation (LoRA) in the image encoder. Additionally, the final block of the encoder is unfrozen to enhance discriminative feature extraction, helping the model distinguish objects from complex backgrounds, such as a plane from a boarding bridge.
Optimized Mask Decoder
The model integrates early-stage texture features with late-stage global context, building upon the HQ-SAM architecture. Unlike previous approaches that freeze the SAM decoder, ROS-SAM employs an alternating optimization strategy to update both the original and high-quality mask branches simultaneously.
Redesigned Data Pipeline
During training, Large-Scale Jittering (LSJ) and random rotations are used to improve scale and orientation invariance. For inference, a "crop-and-upsample" strategy is implemented. Instead of resizing the entire high-resolution satellite image, the model crops a patch around the user's prompt and upsamples it by 2x, ensuring the model sees fine details.
Experimental Results
ROS-SAM was evaluated on the SAT-MTB benchmark, as well as static datasets like iSAID and NPWS VHR-10.
- Performance Boost: ROS-SAM achieved a 13% IoU increase over the original SAM on the SAT-MTB dataset.
- Pipeline Impact: The redesigned data pipeline (LSJ and cropping) alone contributed a 6% boost in IoU.
- Generalization: On the iSAID dataset, ROS-SAM reached 73.22% IoU, significantly outperforming HQ-SAM (63.96%) and the original SAM (53.19%).
Zero-Shot Capabilities
The model was tested on tracking-only datasets such as SatSOT, VISO, and OOTB. Despite never seeing these datasets during training, ROS-SAM generated high-quality masks comparable to manual annotations, demonstrating powerful zero-shot capabilities.
Conclusion
ROS-SAM successfully adapts interactive segmentation for the unique requirements of remote sensing. By focusing on resolution preservation and domain-specific fine-tuning, it provides a powerful tool for generating high-quality segmentation masks from tracking data. While the model still faces challenges with extremely fine details or low-contrast objects, future work will explore leveraging temporal consistency to move toward fully automated video segmentation.
Summary updated