Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation

Yue Han, Jiangning Zhang, Yabiao Wang, Chengjie Wang, Yong Liu, Lu Qi, Ming-Hsuan Yang, Xiangtai Li

Abstract

Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited support examples. We introduce Reference Twice (RefT), a simple yet unified framework that explores the relationship between support and query features using a Transformer-based architecture. RefT enhances query features twice: first at the feature level using mask-based dynamic weighting, and second at the instance level by linking support object queries via cross-attention. Our method achieves state-of-the-art results on COCO, boosting nAP by +8.2 for 10-shot FSIS.

Introduction

Few-Shot Instance Segmentation (FSIS) is a critical "cold start" problem in computer vision: detecting and segmenting brand-new object classes using only a handful of labeled examples. Traditional methods typically rely on Region Proposal Networks (RPNs). However, these architectures face two major challenges:

  1. Supervision Collapse: RPNs trained on base classes often learn to suppress novel classes as background.
  2. Spatial Information Loss: Dual-branch models often compress support features into global "class prototypes," destroying the spatial details necessary for precise segmentation.

To overcome these, we propose Reference Twice (RefT), a framework built on the Mask2Former transformer architecture that handles novel classes flexibly without losing base class knowledge.

The Reference Twice Framework

RefT leverages a "two-stage" reference mechanism to fully exploit the relationship between support and query features.

1. Feature-Level Enhancement (First Reference)

The model extracts multi-scale features from support images. Using "mask pooling" guided by the ground truth support masks, it generates adaptive class prototypes. These prototypes are then used to re-weight and filter query image features, highlighting regions likely to contain the target class.

2. Query-Level Enhancement (Second Reference)

A key insight of this work is that object queries in the support branch already encode high-quality localization and category information. By using cross-attention to link these support object queries directly to the query image's object queries, the model "calibrates" its predictions. This helps distinguish between semantically similar classes, such as cows and horses.

The Reference Twice framework (bottom) compared to traditional RPN-based dual-branch frameworks (top).

For Incremental FSIS (iFSIS), where models must learn new classes without forgetting old ones, we identify the Class Adaptation Layer (CAL) as crucial. We introduce a Class-Enhanced Base Knowledge Distillation (CE-BCD) loss to force the model to retain base knowledge by distilling features from background and base-class regions while masking novel objects.

Experimental Results

RefT was evaluated on MS-COCO 2014, LVIS, and COCO2FSS-1000. It demonstrates a massive performance leap over previous state-of-the-art (SOTA) methods.

  • COCO 10-shot: Achieved 18.4 nAP, an +8.2 point gain over previous RPN-based SOTA.
  • COCO 30-shot: Achieved 22.5 nAP, a +9.4 point gain.
  • Generalized FSIS (gFSIS): RefT maintains high performance on base classes (36.0 bAP) while excelling at novel ones, proving it avoids catastrophic forgetting.
Performance comparison on COCO dataset across FSIS, gFSIS, and iFSIS settings.

RefT is also highly efficient. Unlike meta-learning models that require separate forward passes for every class, RefT handles all classes simultaneously in a single pass, achieving an inference speed of approximately 0.11s per image on an RTX 3090.

Conclusion

Reference Twice provides a simple, unified, and powerful baseline for few-shot vision tasks. By moving away from RPNs to a query-based transformer structure and utilizing a dual-stage reference mechanism, it significantly improves the accuracy and efficiency of few-shot instance segmentation. Future work will focus on improving 1-shot performance and developing more sophisticated object matching logic.

Summary updated

Abstract

Few-Shot Instance Segmentation (FSIS) requires detecting and segmenting novel classes with limited support examples. Existing methods based on Region Proposal Networks (RPNs) face two issues: 1) overfitting suppresses novel class objects and 2) dual-branch models require complex spatial correlation strategies to prevent spatial information loss when generating class prototypes. We introduce a unified framework, Ref erence T wice (RefT), to exploit the relationship between support and query features for FSIS and related tasks. Our three main contributions are: 1) a novel transformer-based baseline that avoids overfitting, offering a new direction for FSIS; 2) demonstrating that support object queries encode key factors after base training, allowing query features to be enhanced twice at both feature and query levels using simple cross-attention, thus avoiding complex spatial correlation interaction; and 3) introducing a class-enhanced base knowledge distillation loss to address the issue of DETR-like models struggling with incremental settings due to the input projection layer, enabling easy extension to incremental FSIS. Extensive experimental evaluations on the COCO dataset under three FSIS settings demonstrate that our method performs favorably against existing approaches across different shots, e.g., +8.2/+9.4 performance gain over state-of-the-art methods with 10/30-shots.
BibTeX
@article{han2023reference,
  author = {Yue Han and Jiangning Zhang and Yabiao Wang and Chengjie Wang and Yong Liu and Lu Qi and Ming-Hsuan Yang and Xiangtai Li},
  title = {{Reference Twice: A Simple and Unified Baseline for Few-Shot Instance Segmentation}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2023},
  url = {https://openreview.net/forum?id=Xvx9gWw8ks},
}
Figure