Introduction
The Segment Anything Model (SAM) has set a new standard for interactive segmentation in natural images. However, applying SAM to remote sensing (RS) data reveals significant limitations. Satellite imagery presents unique challenges: objects are often tiny (scale/resolution), orientations are arbitrary (lack of a "gravity" prior), and there is a substantial domain gap between natural photos and overhead views. For instance, SAM often fails to distinguish between a plane and its boarding bridge or loses small objects entirely during its standard 1024x1024 resizing process.
The ROS-SAM Framework
To address these issues, the authors propose ROS-SAM, a specialized adaptation of SAM for remote sensing moving objects. The architecture incorporates three primary innovations:
- LoRA-based Fine-tuning: Instead of retraining the entire Vision Transformer (ViT) encoder, ROS-SAM utilizes Low-Rank Adaptation (LoRA) on the Query and Value projection matrices. This allows the model to learn RS-specific features efficiently. Additionally, the final block of the image encoder is unfrozen to enhance discriminative power.
- Hybrid Mask Decoder: Integrating concepts from HQ-SAM, ROS-SAM adds a high-quality decoder branch that fuses early local features with final global context. An alternating optimization strategy is used during training to refine mask quality while preserving SAM's original zero-shot capabilities.
- Optimized Data Pipeline: For training, Large-Scale Jittering (LSJ) and random rotations are employed to handle scale and orientation variance. During inference, a center-cropping strategy based on user prompts, followed by 2x bicubic upsampling, ensures that fine-grained details are preserved.
Experimental Results
ROS-SAM was evaluated on several benchmarks, including SAT-MTB (remote sensing video) and iSAID (static RS images).
Performance Gains
The model achieved a 13% IoU improvement over the original SAM on the SAT-MTB dataset. A significant portion of this gain (6%) is attributed to the redesigned data pipeline, specifically the use of random rotations to overcome orientation bias. On the iSAID dataset, ROS-SAM reached a 73.22% IoU, compared to SAM's 53.19%.
Zero-Shot Generalization
One of the most impressive findings is ROS-SAM's performance on tracking datasets like SatSOT and VISO. Despite only being provided with bounding boxes, the model generated high-quality masks, effectively transforming tracking data into segmentation data. Unlike the original SAM, which often produced generic "star" shapes for complex objects like airplanes, ROS-SAM captured precise boundaries.
Conclusion and Future Work
ROS-SAM demonstrates that targeted architectural modifications and domain-specific data pipelines can bridge the gap between general-purpose foundation models and specialized remote sensing tasks. While the model significantly improves mask quality for tiny moving objects, future work will focus on integrating temporal consistency to enable fully automated video object segmentation across consecutive frames.
Summary updated