Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation

Chunbo Lang, Binfei Tu, Gong Cheng, Junwei Han

TL;DR

This paper introduces "Divide-and-Conquer Proxies" (DCP), a novel framework for few-shot segmentation that moves beyond traditional prototype-based methods. By generating a set of specialized "proxies" from support images, DCP provides more robust and reliable guidance for segmenting unseen objects, achieving state-of-the-art results on standard benchmarks.

What Problem Does This Paper Address?

Few-shot segmentation (FSS) aims to identify and segment objects of new categories using only a few labeled examples (support set). This is crucial because acquiring large annotated datasets for every possible object category is prohibitively expensive. Existing FSS methods often rely on "prototype learning," where a single or a few representative features (prototypes) are extracted from the support set to guide the segmentation of a query image. However, this approach can be fragile. It struggles with ambiguous object boundaries, incomplete object segmentation, and can be easily misled by irrelevant objects in the scene. The authors argue that these failures stem from an incomplete utilization of the information present in the support image-mask pairs, leading to insufficient guidance.

Methods

The core innovation of DCP lies in its "divide-and-conquer" strategy. Instead of relying on a single prototype, DCP first employs a "self-reasoning scheme" on the support image to generate a coarse segmentation mask. This mask is then divided into multiple regions with distinct properties (e.g., main foreground, boundary foreground, main background, distractor background).

These distinct regions are then used to derive a set of specialized "proxies" through masked average pooling on the support features. Each proxy is designed to capture specific aspects of the scene, such as foreground cues, ambiguous boundaries, or background distractions. This is the "divide" part.

The "conquer" part involves a novel "parallel decoder structure" (PDS). This structure integrates proxies with similar attributes, allowing them to collectively boost the discrimination power for the query image. This is a departure from simply combining all information; instead, it intelligently groups related guidance signals. The authors also introduce auxiliary losses to ensure the self-reasoning scheme provides reliable information.

Datasets

The paper evaluates DCP on two standard few-shot segmentation benchmarks:

  • PASCAL-5²: Derived from PASCAL VOC 2012, with additional annotations.
  • COCO-20²: Built from MS COCO.

Both datasets are divided into four folds for cross-validation, with models trained on three folds and tested on the remaining one.

Key Insights and Ideas

  • Beyond the Prototype: The central idea is that a single prototype is insufficient. By creating multiple, specialized proxies, the model can address different segmentation challenges more effectively.
  • Divide and Conquer for Segmentation: The framework explicitly divides the problem (segmentation mask) and then conquers specific challenges (ambiguous boundaries, distractors) with tailored proxies.
  • Auxiliary Information is Key: The self-reasoning scheme and the derived masks provide valuable auxiliary information that goes beyond just object features, leading to more robust guidance.
  • Intelligent Fusion: The parallel decoder structure is crucial for effectively combining these diverse proxies, preventing information confusion that might arise from simpler fusion methods.
  • Background Proxies Matter: Surprisingly, background-related proxies (like Mδ) proved effective, suggesting that explicitly handling background and distractor regions can significantly improve segmentation performance.

Key Results

DCP achieves state-of-the-art results on both PASCAL-5² and COCO-20² datasets.

  • On PASCAL-5², DCP outperforms conventional prototype-based approaches by up to 5-10% on average.
  • On COCO-20², DCP shows improvements of 2.89% (1-shot) and 3.78% (5-shot) in mIoU over previous best methods.
  • Qualitative results show a significant reduction in false positives caused by irrelevant objects and more accurate segmentation boundaries.
  • The proposed parallel decoder structure (PDS) is shown to be more effective than simpler fusion strategies, achieving better performance with fewer parameters and faster inference speeds.

Limitations and Future Work

  • Computational Overhead: While DCP shows efficiency gains with its PDS, the generation of multiple proxies and their integration still adds computational complexity compared to very basic methods.
  • Generalization to More Complex Scenes: While improvements are significant, the paper focuses on standard benchmarks. Further evaluation on more diverse and challenging datasets could reveal limitations.
  • Future Work: The authors suggest that the "beyond the prototype" concept could be further explored. The effectiveness of background proxies also opens avenues for more sophisticated background modeling in FSS. The proposed framework also offers a strong baseline for future research in few-shot segmentation.

Summary updated

Abstract

Few-shot segmentation, which aims to segment unseen-class objects given only a handful of densely labeled samples, has received widespread attention from the community. Existing approaches typically follow the prototype learning paradigm to perform meta-inference, which fails to fully exploit the underlying information from support image-mask pairs, resulting in various segmentation failures, e.g., incomplete objects, ambiguous boundaries, and distractor activation. To this end, we propose a simple yet versatile framework in the spirit of divide-and-conquer. Specifically, a novel self-reasoning scheme is first implemented on the annotated support image, and then the coarse segmentation mask is divided into multiple regions with different properties. Leveraging effective masked average pooling operations, a series of support-induced proxies are thus derived, each playing a specific role in conquering the above challenges. Moreover, we devise a unique parallel decoder structure that integrates proxies with similar attributes to boost the discrimination power. Our proposed approach, named divide-and-conquer proxies (DCP), allows for the development of appropriate and reliable information as a guide at the “episode” level, not just about the object cues themselves. Extensive experiments on PASCAL-5i and COCO-20i demonstrate the superiority of DCP over conventional prototype-based approaches (up to 5~10% on average), which also establishes a new state-of-the-art. Code is available at github.com/chunbolang/DCP.
BibTeX
@inproceedings{lang2022beyond,
  author = {Chunbo Lang and Binfei Tu and Gong Cheng and Junwei Han},
  title = {{Beyond the Prototype: Divide-and-conquer Proxies for Few-shot Segmentation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/143},
}
Figure