Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation

Chunbo Lang, Binfei Tu, Gong Cheng, Junwei Han

Abstract

Few-shot segmentation (FSS) aims to segment unseen-class objects given only a handful of densely labeled samples. Existing approaches typically follow the prototype learning paradigm to perform meta-inference, which fails to fully exploit the underlying information from support image-mask pairs, resulting in various segmentation failures, such as incomplete objects, ambiguous boundaries, and distractor activation. To this end, we propose a simple yet versatile framework in the spirit of divide-and-conquer. Specifically, a novel self-reasoning scheme is first implemented on the annotated support image, and then the coarse segmentation mask is divided into multiple regions with different properties. Leveraging effective masked average pooling operations, a series of support-induced proxies are derived, each playing a specific role in conquering the above challenges. Moreover, we devise a unique parallel decoder structure that integrates proxies with similar attributes to boost the discrimination power. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information as a guide at the "episode" level, not just about the object cues themselves. Extensive experiments on PASCAL-5i and COCO-20i demonstrate the superiority of DCP over conventional prototype-based approaches (up to 5~10% on average), establishing a new state-of-the-art.

Introduction

Few-shot segmentation is a challenging task that requires models to generalize to new categories using minimal supervision. The dominant prototype learning paradigm typically compresses foreground features from a support image into a single vector. However, this "squeezing" process is inherently lossy, often failing to capture the complexity of object boundaries or the presence of similar-looking distractors in the background.

Conceptual comparison between conventional prototype learning and the proposed Divide-and-Conquer Proxies (DCP) framework.

As shown in Figure 1, conventional methods often result in incomplete segmentation or confusion with background elements. We address these issues by moving beyond a single prototype to a set of specialized "proxies" that explicitly model different aspects of the scene.

Methodology

1. The "Divide" Phase: Proxy Acquisition

Instead of relying solely on the provided ground-truth mask, DCP employs a self-reasoning scheme. The model first attempts to segment the support image itself. By comparing this prediction with the ground truth, the support image is decomposed into four distinct regions:

  • $\alpha$ (Main Foreground): Regions correctly identified as foreground.
  • $\beta$ (Boundary/Hard Foreground): Foreground regions missed by the model (e.g., thin parts or edges).
  • $\gamma$ (Primary Background): Regions correctly identified as background.
  • $\delta$ (Distractors): Background regions mistakenly identified as foreground.

Feature vectors, or proxies, are then extracted from these regions using masked average pooling, providing a multi-faceted representation of the target class and its environment.

2. The "Conquer" Phase: Parallel Decoder Structure (PDS)

To effectively utilize these proxies, we introduce a Parallel Decoder Structure. Simply combining all proxies into a single stream can lead to information confusion. Instead, our PDS uses two parallel branches:

  • Foreground Information Flow: Utilizes $\alpha$ and $\beta$ proxies to refine object shape and boundaries.
  • Background Information Flow: Utilizes $\gamma$ and $\delta$ proxies to suppress distractors and clarify the background.
The DCP architecture, showing the feature extraction process and the Parallel Decoder Structure (PDS) for foreground and background information flow.

Experimental Results

DCP was evaluated on two major benchmarks, PASCAL-5i and COCO-20i, where it established new state-of-the-art results. On PASCAL-5i, the model achieved a mean Intersection over Union (mIoU) of 61.31% in the 1-shot setting and 65.84% in the 5-shot setting using a VGG16 backbone. On the more complex COCO-20i dataset, DCP outperformed the previous best model (PFENet) by 2.89% (1-shot) and 3.78% (5-shot).

Qualitative results indicate that DCP is particularly effective at ignoring "distractor" objects—such as a cat when the target is a dog—and producing sharper, more accurate boundaries for complex shapes. The model maintains high efficiency, operating at over 16 FPS.

Conclusion

The Divide-and-Conquer Proxies (DCP) framework offers a robust alternative to standard prototype learning for few-shot segmentation. By leveraging the model's own reasoning failures to identify hard foreground and distractor regions, DCP provides superior guidance for segmenting unseen objects in complex scenes. Future work may extend this logic to related tasks like video object segmentation.

Summary updated

Abstract

Few-shot segmentation, which aims to segment unseen-class objects given only a handful of densely labeled samples, has received widespread attention from the community. Existing approaches typically follow the prototype learning paradigm to perform meta-inference, which fails to fully exploit the underlying information from support image-mask pairs, resulting in various segmentation failures, e.g., incomplete objects, ambiguous boundaries, and distractor activation. To this end, we propose a simple yet versatile framework in the spirit of divide-and-conquer. Specifically, a novel self-reasoning scheme is first implemented on the annotated support image, and then the coarse segmentation mask is divided into multiple regions with different properties. Leveraging effective masked average pooling operations, a series of support-induced proxies are thus derived, each playing a specific role in conquering the above challenges. Moreover, we devise a unique parallel decoder structure that integrates proxies with similar attributes to boost the discrimination power. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information as a guide at the "episode" level, not just about the object cues themselves. Extensive experiments on PASCAL-5i and COCO-20i demonstrate the superiority of DCP over conventional prototype-based approaches (up to 5~10% on average), which also establishes a new state-of-the-art. Code is available at github.com/chunbolang/DCP.
BibTeX
@inproceedings{lang2022beyond,
  author = {Chunbo Lang and Binfei Tu and Gong Cheng and Junwei Han},
  title = {{Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/143},
}
Figure