Abstract
Few Shot Instance Segmentation (FSIS) requires models to detect and segment novel classes with limited support examples. We introduce Reference Twice (RefT), a simple yet unified framework that explores the relationship between support and query features using a Transformer-based architecture. RefT enhances query features twice: first at the feature level using mask-based dynamic weighting, and second at the instance level by linking support object queries via cross-attention. Our method achieves state-of-the-art results on COCO, boosting nAP by +8.2 for 10-shot FSIS.
Introduction
Few-Shot Instance Segmentation (FSIS) is a critical "cold start" problem in computer vision: detecting and segmenting brand-new object classes using only a handful of labeled examples. Traditional methods typically rely on Region Proposal Networks (RPNs). However, these architectures face two major challenges:
- Supervision Collapse: RPNs trained on base classes often learn to suppress novel classes as background.
- Spatial Information Loss: Dual-branch models often compress support features into global "class prototypes," destroying the spatial details necessary for precise segmentation.
To overcome these, we propose Reference Twice (RefT), a framework built on the Mask2Former transformer architecture that handles novel classes flexibly without losing base class knowledge.
The Reference Twice Framework
RefT leverages a "two-stage" reference mechanism to fully exploit the relationship between support and query features.
1. Feature-Level Enhancement (First Reference)
The model extracts multi-scale features from support images. Using "mask pooling" guided by the ground truth support masks, it generates adaptive class prototypes. These prototypes are then used to re-weight and filter query image features, highlighting regions likely to contain the target class.
2. Query-Level Enhancement (Second Reference)
A key insight of this work is that object queries in the support branch already encode high-quality localization and category information. By using cross-attention to link these support object queries directly to the query image's object queries, the model "calibrates" its predictions. This helps distinguish between semantically similar classes, such as cows and horses.
For Incremental FSIS (iFSIS), where models must learn new classes without forgetting old ones, we identify the Class Adaptation Layer (CAL) as crucial. We introduce a Class-Enhanced Base Knowledge Distillation (CE-BCD) loss to force the model to retain base knowledge by distilling features from background and base-class regions while masking novel objects.
Experimental Results
RefT was evaluated on MS-COCO 2014, LVIS, and COCO2FSS-1000. It demonstrates a massive performance leap over previous state-of-the-art (SOTA) methods.
- COCO 10-shot: Achieved 18.4 nAP, an +8.2 point gain over previous RPN-based SOTA.
- COCO 30-shot: Achieved 22.5 nAP, a +9.4 point gain.
- Generalized FSIS (gFSIS): RefT maintains high performance on base classes (36.0 bAP) while excelling at novel ones, proving it avoids catastrophic forgetting.
RefT is also highly efficient. Unlike meta-learning models that require separate forward passes for every class, RefT handles all classes simultaneously in a single pass, achieving an inference speed of approximately 0.11s per image on an RTX 3090.
Conclusion
Reference Twice provides a simple, unified, and powerful baseline for few-shot vision tasks. By moving away from RPNs to a query-based transformer structure and utilizing a dual-stage reference mechanism, it significantly improves the accuracy and efficiency of few-shot instance segmentation. Future work will focus on improving 1-shot performance and developing more sophisticated object matching logic.
Summary updated