TL;DR
This paper investigates the performance of the "Segment Anything Model" (SAM), a powerful foundation model for image segmentation, on overhead (satellite) imagery. While SAM excels at segmenting objects in natural images, this study reveals its performance can be inconsistent on satellite data, highlighting specific failure cases and suggesting avenues for improvement.
What Problem Does This Paper Address?
Foundation models, like SAM, are trained on massive datasets and can generalize to new tasks with little to no additional training (zero-shot learning). SAM, in particular, has shown impressive zero-shot segmentation capabilities on natural imagery, significantly reducing the effort required for tasks like image annotation. However, overhead imagery presents unique challenges compared to natural imagery: objects are often larger-scale, viewed from a different perspective, and can appear at any orientation. It's unclear if SAM's impressive generalization extends to this domain. This paper aims to answer: "Can SAM segment anything from space?" by evaluating its performance on various overhead imagery tasks and identifying its limitations.
Methods
The authors replicate two key application scenarios for SAM from its original publication:
- Model Composition: SAM is used to refine or generate segmentation masks based on prompts from another vision model (e.g., a bounding box from a detector). Here, they train U-Net models to generate bounding box prompts and use ground truth labels for point prompts.
- Interactive Annotation: SAM assists human annotators by generating segmentation masks from simple prompts (points or bounding boxes), significantly reducing manual annotation effort. They evaluate SAM's performance with single bounding box prompts, single point prompts (random and center), and iterative point prompts.
They compare SAM's performance against supervised models (U-Net) trained specifically for each task and, in the interactive annotation case, against the RITM (Reviving Iterative Training with Mask Guidance) model.
Datasets
The study utilizes eight public benchmark datasets of overhead imagery, covering diverse object classes (buildings, roads, cloud cover, farming crops), resolutions (0.3m to 30m), and geographic locations. These include:
- Solar
- Inria
- DeepGlobe Roads
- 38-Cloud
- DeepGlobe Land
- Parcel Delineation
- SpaceNet 2
For some datasets, modifications were made, such as combining Inria and DeepGlobe Building datasets (DG+Inria Building) and simplifying DeepGlobe Land into three binary segmentation problems (water, urban, agriculture).
Key Insights and Ideas
- Generalization is Mixed: SAM does generalize to overhead imagery, but its performance is highly variable and often significantly lower than task-specific supervised models, especially for complex objects like buildings and roads.
- Prompt Matters: As with natural imagery, bounding box prompts yield better results than point prompts for SAM in the model composition scenario. However, the performance gap between SAM and supervised models is larger on overhead imagery.
- Interactive Annotation Potential: SAM generally outperforms the RITM baseline for interactive annotation, often by a substantial margin, suggesting its utility for speeding up annotation workflows.
- Resolution Sensitivity: SAM's performance is surprisingly sensitive to image resolution. Artificially upsampling imagery can improve its accuracy, even without adding new information, suggesting a potential bias in its learned object scale.
- Instance Segmentation Challenges: SAM is designed for instance segmentation, but some overhead imagery classes, like roads, are inherently ill-defined as distinct instances, leading to very poor performance.
Key Results
- Model Composition: Supervised models (U-Net SOTA) consistently outperform SAM. SAM's performance is significantly worse on buildings and roads compared to natural imagery benchmarks.
- Interactive Annotation: SAM consistently outperforms RITM, often by a large margin. With 10 point prompts, SAM achieves IoUs greater than 0.75 on 6 out of 10 benchmarks, indicating practical utility.
- Roads are a Major Weakness: SAM performs very poorly on road segmentation tasks across all experiments (IoU < 10), likely due to the ill-defined nature of road instances.
- Solar Arrays are Also Challenging: Despite their apparent simplicity, SAM performs relatively poorly on solar arrays, suggesting generalization issues beyond just complex objects.
Limitations and Future Work
The authors acknowledge that SAM's performance is highly variable on overhead imagery tasks. They hypothesize that fine-tuning SAM's decoder (rather than the entire model) on overhead imagery classes could lead to substantial improvements. They also suggest retraining the decoder for class-level segmentation for classes where instance segmentation is ill-posed (e.g., roads). The sensitivity to resolution also points to potential biases in SAM's training data. Future work could explore these fine-tuning strategies and investigate SAM's behavior on other overhead imagery tasks.
Summary updated