iFS-RCNN: An Incremental Few-shot Instance Segmenter

Khoi Nguyen, Sinisa TodorovicOpenReview 2022 · OpenReview

Abstract

This paper addresses incremental few-shot instance segmentation, where a few examples of new object classes arrive when access to training examples of old classes is not available anymore, and the goal is to perform well on both old and new classes. We make two contributions by extending the common Mask-RCNN framework in its second stage – namely, we specify a new object class classifier based on the probit function and a new uncertainty-guided bounding-box predictor. The former leverages Bayesian learning to address a paucity of training examples of new classes. The latter learns not only to predict object bounding boxes but also to estimate the uncertainty of the prediction as a guidance for bounding box refinement. We also specify two new loss functions in terms of the estimated object-class distribution and bounding-box uncertainty. Our contributions produce significant performance gains on the COCO dataset over the state of the art – specifically, the gain of +6 on the new classes and +16 on the old classes in the AP instance segmentation metric. Furthermore, we are the first to evaluate the incremental few-shot setting on the more challenging LVIS dataset.

Introduction

In computer vision, deep learning models typically suffer from "data hunger," requiring thousands of labeled images to recognize new objects. Incremental Few-Shot Instance Segmentation (iFSIS) addresses the intersection of two critical challenges: learning from very few examples (Few-Shot Learning) and adding new classes over time without access to original training data (Incremental Learning). The primary difficulty in iFSIS is twofold: preventing "catastrophic forgetting" of base classes while avoiding overfitting on the sparse data provided for novel classes.

The iFS-RCNN pipeline: Pre-training on base classes, fine-tuning on new classes, and final testing on both.

Methodology

iFS-RCNN builds upon the standard Mask-RCNN framework, introducing two major architectural innovations to the second-stage heads to handle the scarcity of novel class data and the absence of base class data during fine-tuning.

1. The Probit Classifier

Standard classifiers often exhibit extreme bias when trained on limited samples. iFS-RCNN replaces point-estimate weights with a Bayesian approach, learning a distribution (mean and variance) of class weights. To maintain computational efficiency, the authors utilize a probit function to approximate Bayesian inference analytically, avoiding the need for slow Monte Carlo sampling while retaining the robustness of probabilistic modeling.

2. Uncertainty-Guided Box Predictor

Beyond predicting coordinates, iFS-RCNN estimates the uncertainty for each boundary of the bounding box (Left, Top, Right, Bottom). This uncertainty acts as a functional input for a Refinement Module, allowing the model to adjust boundaries more aggressively where confidence is low. A novel uncertainty-weighted loss function is employed to penalize errors less severely when the model correctly identifies high uncertainty in its predictions.

Detailed architecture of the two main contributions: the Probit Classifier and the Uncertainty-guided Box Predictor.

Architectural Refinements

The framework also transitions from Softmax to Sigmoid activations. While Softmax forces classes to compete—often leading to base-class dominance—Sigmoid treats each class independently, significantly reducing bias toward data-rich base classes.

Experiments and Results

The model was evaluated on MS-COCO (60 base, 20 novel classes) and the long-tailed LVIS dataset, which contains over 1,200 classes. iFS-RCNN established a new state-of-the-art benchmark for incremental few-shot settings on LVIS.

On the COCO dataset, iFS-RCNN achieved substantial performance gains over the previous state-of-the-art (iMTFA):

  • Base Classes: A massive +16 AP improvement, demonstrating superior retention of original knowledge.
  • Novel Classes: A gain of +6 AP, proving the effectiveness of the Bayesian Probit head in low-data regimes.

Qualitative analysis shows that the uncertainty-guided refinement leads to tighter, more accurate segmentation masks in cluttered scenes compared to standard baselines.

Conclusion

iFS-RCNN successfully mitigates forgetting and overfitting in incremental few-shot scenarios. While currently optimized for RPN-based detectors like Mask-RCNN, the framework demonstrates that treating uncertainty as a functional feature and adopting Bayesian classifiers can yield dramatic improvements in model robustness. Future work may explore extending these principles to transformer-based architectures and multi-stage incremental learning scenarios.

Summary updated

Abstract

This paper addresses incremental few-shot instance segmentation, where a few examples of new object classes arrive when access to training examples of old classes is not available anymore, and the goal is to perform well on both old and new classes. We make two contributions by extending the common Mask-RCNN framework in its second stage -- namely, we specify a new object class classifier based on the probit function and a new uncertainty-guided bounding-box predictor. The former leverages Bayesian learning to address a paucity of training examples of new classes. The latter learns not only to predict object bounding boxes but also to estimate the uncertainty of the prediction as guidance for bounding box refinement. We also specify two new loss functions in terms of the estimated object-class distribution and bounding-box uncertainty. Our contributions produce significant performance gains on the COCO dataset over the state of the art -- specifically, the gain of +6 on the new classes and +16 on the old classes in the AP instance segmentation metric. Furthermore, we are the first to evaluate the incremental few-shot setting on the more challenging LVIS dataset.
BibTeX
@misc{nguyen2022ifs,
  author = {Khoi Nguyen and Sinisa Todorovic},
  title = {{iFS-RCNN: An Incremental Few-shot Instance Segmenter}},
  year = {2022},
  eprint = {2205.15562},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.15562},
  howpublished = {OpenReview},
}
Figure