APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation

Weizhao He, Yang Zhang, Wei Zhuo, Linlin Shen, Jiaqi Yang, Songhe Deng, Liang Sun

Abstract

Few-shot semantic segmentation (FSS) aims to segment novel objects using minimal labeled data. However, existing models often struggle with cross-domain (CD-FSS) scenarios where the target domain differs significantly from the training data. This paper introduces APSeg, an auto-prompt network that adapts the Segment Anything Model (SAM) for CD-FSS. APSeg eliminates the need for manual prompts and target-domain fine-tuning by introducing two key components: Dual Prototype Anchor Transformation (DPAT) for domain-agnostic feature alignment and a Meta Prompt Generator (MPG) for automated prompt synthesis. Experimental results demonstrate that APSeg achieves state-of-the-art performance across diverse domains, including medical and satellite imagery.

Introduction

Semantic segmentation is a cornerstone of computer vision, yet its reliance on large-scale annotated datasets limits its applicability in niche domains. Few-shot semantic segmentation (FSS) addresses this by learning from a few examples. However, a significant challenge remains: cross-domain generalization. A model trained on natural images (e.g., PASCAL VOC) typically fails when applied to specialized domains like medical X-rays or remote sensing.

While foundation models like the Segment Anything Model (SAM) exhibit impressive zero-shot capabilities, they are hindered by a dependency on manual interactive prompts (clicks or boxes) and performance degradation in specialized domains. APSeg bridges this gap by automating the prompting process and aligning features into a domain-independent space.

Methods

The APSeg framework maintains the core components of SAM—the image encoder and mask decoder—in a frozen state, training only lightweight adapter modules. This approach leverages SAM's rich internal representations while specializing its output for few-shot tasks.

The APSeg framework architecture, featuring the Dual Prototype Anchor Transformation (DPAT) and Meta Prompt Generator (MPG) modules.

Dual Prototype Anchor Transformation (DPAT)

Traditional feature transformation methods often rely solely on the support image. DPAT improves upon this by incorporating "cycle-consistency" to derive pseudo query prototypes. By matching query features back to the support set and validating the correspondence, the model establishes a stable, domain-agnostic feature space. This ensures that the relationship between foreground and background remains consistent regardless of the visual domain.

Meta Prompt Generator (MPG)

To remove the "human-in-the-loop" requirement of SAM, APSeg utilizes the Meta Prompt Generator. This module automatically synthesizes both sparse (point-like) and dense (mask-like) prompt embeddings directly from the transformed image features. These meta-prompts are then fed into SAM’s decoder, allowing the system to identify and segment specific semantics autonomously.

Experimental Setup

APSeg was trained on the PASCAL VOC 2012 dataset and evaluated on four distinct target domains to assess its cross-domain robustness:

  • Chest X-ray: Lung segmentation in radiographs.
  • ISIC: Identification of skin lesions in dermoscopic images.
  • FSS-1000: A diverse set of 1,000 natural object categories.
  • Deepglobe: Land cover classification in high-resolution satellite imagery.

Key Results

APSeg establishes a new state-of-the-art for CD-FSS, outperforming previous methods like PATNet by an average of 5.24% in 1-shot and 3.10% in 5-shot settings.

Qualitative segmentation results across four target domains: Chest X-ray, ISIC, FSS-1000, and Deepglobe.

The most significant improvements were observed in the medical domain. On the Chest X-ray dataset, APSeg achieved a staggering 17.49% increase in mIoU over existing benchmarks. Furthermore, APSeg significantly outperformed PerSAM, a SAM-based method using heuristic prompts, demonstrating that learned meta-prompts are superior for complex cross-domain tasks.

Conclusion

APSeg demonstrates that foundation models like SAM can be effectively adapted for specialized, low-data tasks through intelligent auto-prompting and feature alignment. While the model is highly efficient with only 0.7M learnable parameters, its reliance on the large SAM backbone remains a consideration for deployment on resource-constrained devices. Future research will explore the use of smaller foundation models and dynamic prompt scaling to further optimize performance and efficiency.

Summary updated

Abstract

Few-shot semantic segmentation (FSS) endeavors to segment unseen classes with only a few labeled samples. Current FSS methods are commonly built on the assumption that their training and application scenarios share similar domains, and their performances degrade significantly while applied to a distinct domain. To this end, we propose to leverage the cutting-edge foundation model, the segment Anything Model (SAM), for generalization enhancement. The SAM however performs unsatisfactorily on domains that are distinct from its training data, which primarily comprise natural scene images, and it does not support automatic segmentation of specific semantics due to its interactive prompting mechanism. In our work, we introduce APSeg, a novel auto-prompt network for cross-domain few-shot semantic segmentation (CD-FSS), which is designed to be auto-prompted for guiding cross-domain segmentation. Specifically, we propose a Dual Prototype Anchor Transformation (DPAT) module that fuses pseudo query prototypes extracted based on cycle-consistency with support prototypes, allowing features to be transformed into a more stable domain-agnostic space. Additionally, a Meta Prompt (MPG) module is introduced to automatically generate prompt embeddings, eliminating the need for manual visual prompts. We build an efficient model which can be applied directly to target domains without fine-tuning. Extensive experiments on four cross-domain datasets show that our model outperforms the state-of-the-art CD-FSS method by 5.24% and 3.10% in average accuracy on 1-shot and 5-shot settings, respectively.
BibTeX
@misc{he2024apseg,
  author = {Weizhao He and Yang Zhang and Wei Zhuo and Linlin Shen and Jiaqi Yang and Songhe Deng and Liang Sun},
  title = {{APSeg: Auto-Prompt Network for Cross-Domain Few-Shot Semantic Segmentation}},
  year = {2024},
  eprint = {2406.08372},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2406.08372},
}
Figure