Abstract
Few-shot semantic segmentation (FSS) aims to segment novel objects using minimal labeled data. However, existing models often struggle with cross-domain (CD-FSS) scenarios where the target domain differs significantly from the training data. This paper introduces APSeg, an auto-prompt network that adapts the Segment Anything Model (SAM) for CD-FSS. APSeg eliminates the need for manual prompts and target-domain fine-tuning by introducing two key components: Dual Prototype Anchor Transformation (DPAT) for domain-agnostic feature alignment and a Meta Prompt Generator (MPG) for automated prompt synthesis. Experimental results demonstrate that APSeg achieves state-of-the-art performance across diverse domains, including medical and satellite imagery.
Introduction
Semantic segmentation is a cornerstone of computer vision, yet its reliance on large-scale annotated datasets limits its applicability in niche domains. Few-shot semantic segmentation (FSS) addresses this by learning from a few examples. However, a significant challenge remains: cross-domain generalization. A model trained on natural images (e.g., PASCAL VOC) typically fails when applied to specialized domains like medical X-rays or remote sensing.
While foundation models like the Segment Anything Model (SAM) exhibit impressive zero-shot capabilities, they are hindered by a dependency on manual interactive prompts (clicks or boxes) and performance degradation in specialized domains. APSeg bridges this gap by automating the prompting process and aligning features into a domain-independent space.
Methods
The APSeg framework maintains the core components of SAM—the image encoder and mask decoder—in a frozen state, training only lightweight adapter modules. This approach leverages SAM's rich internal representations while specializing its output for few-shot tasks.
Dual Prototype Anchor Transformation (DPAT)
Traditional feature transformation methods often rely solely on the support image. DPAT improves upon this by incorporating "cycle-consistency" to derive pseudo query prototypes. By matching query features back to the support set and validating the correspondence, the model establishes a stable, domain-agnostic feature space. This ensures that the relationship between foreground and background remains consistent regardless of the visual domain.
Meta Prompt Generator (MPG)
To remove the "human-in-the-loop" requirement of SAM, APSeg utilizes the Meta Prompt Generator. This module automatically synthesizes both sparse (point-like) and dense (mask-like) prompt embeddings directly from the transformed image features. These meta-prompts are then fed into SAM’s decoder, allowing the system to identify and segment specific semantics autonomously.
Experimental Setup
APSeg was trained on the PASCAL VOC 2012 dataset and evaluated on four distinct target domains to assess its cross-domain robustness:
- Chest X-ray: Lung segmentation in radiographs.
- ISIC: Identification of skin lesions in dermoscopic images.
- FSS-1000: A diverse set of 1,000 natural object categories.
- Deepglobe: Land cover classification in high-resolution satellite imagery.
Key Results
APSeg establishes a new state-of-the-art for CD-FSS, outperforming previous methods like PATNet by an average of 5.24% in 1-shot and 3.10% in 5-shot settings.
The most significant improvements were observed in the medical domain. On the Chest X-ray dataset, APSeg achieved a staggering 17.49% increase in mIoU over existing benchmarks. Furthermore, APSeg significantly outperformed PerSAM, a SAM-based method using heuristic prompts, demonstrating that learned meta-prompts are superior for complex cross-domain tasks.
Conclusion
APSeg demonstrates that foundation models like SAM can be effectively adapted for specialized, low-data tasks through intelligent auto-prompting and feature alignment. While the model is highly efficient with only 0.7M learnable parameters, its reliance on the large SAM backbone remains a consideration for deployment on resource-constrained devices. Future research will explore the use of smaller foundation models and dynamic prompt scaling to further optimize performance and efficiency.
Summary updated