Prediction Calibration for Generalized Few-shot Semantic Segmentation

Zhihe Lu, Sen He, Da Li, Yi-Zhe Song, Tao Xiang

Abstract

Generalized Few-shot Semantic Segmentation (GFSS) aims to segment each image pixel into either base classes with abundant training examples or novel classes with only a handful of training images per class. Compared to the widely studied Few-shot Semantic Segmentation (FSS), which is limited to segmenting novel classes only, GFSS is much under-studied despite being more practical. Existing approaches to GFSS are based on classifier parameter fusion whereby a newly trained novel class classifier and a pre-trained base class classifier are combined. As the training data is dominated by base classes, this approach is inevitably biased towards the base classes. In this work, we propose a novel Prediction Calibration Network (PCN) to address this problem. Instead of fusing the classifier parameters, we fuse the scores produced separately by the base and novel classifiers. To ensure that the fused scores are not biased, a new Transformer-based calibration module is introduced. This module uses feature-score cross-covariance and is episodically trained to be generalizable. Extensive experiments on PASCAL-$5^{i}$ and COCO-$20^{i}$ show that our PCN outperforms the state-of-the-art alternatives by large margins.

Introduction

In realistic computer vision applications, new objects (novel classes) typically appear alongside well-known objects (base classes). While standard Few-shot Semantic Segmentation (FSS) focuses on identifying novel objects using only 1–5 examples, it ignores the base classes during inference. Generalized Few-shot Semantic Segmentation (GFSS) addresses this by requiring the model to segment both base and novel classes simultaneously.

The primary challenge in GFSS is the extreme data imbalance. Base classes have thousands of training samples, while novel classes have very few. Traditional methods that merge classifier weights often result in a heavy bias toward base classes. Conversely, attempts to boost novel class detection can lead to "overconfidence," where base-class pixels are incorrectly labeled as novel ones.

Methodology: Prediction Calibration Network (PCN)

The proposed PCN architecture addresses these challenges through a two-stage fusion and calibration process.

Overview of the PCN architecture featuring Scores Prediction, Normalized Score Fusion, and Transformer-based Calibration.

Normalized Score Fusion (NSF)

Instead of merging internal classifier parameters, PCN runs base and novel classifiers separately. It applies a softmax operation to each set of scores independently to normalize them before fusion. This approach significantly reduces the initial bias toward base classes that occurs when weights are merged directly.

Transformer-based Calibration

To resolve the "overconfidence trap" where novel classes misidentify base-class pixels, PCN introduces a Transformer-based calibration module.

  • Cross-covariance Attention: Standard Transformers are computationally expensive for pixel-level tasks. PCN treats class scores and feature channels as tokens, computing the covariance between them to maintain efficiency while capturing global context.
  • Multi-level Feature Guidance: The module utilizes both high-level semantic features and low-level edge information to ensure that final segmentation masks respect object boundaries and spatial details.

Results and Discussion

PCN was evaluated on the PASCAL-$5^i$ and COCO-$20^i$ benchmarks, demonstrating significant improvements over existing state-of-the-art (SOTA) methods.

Qualitative comparison of confidence maps and final predictions, demonstrating PCN's ability to mitigate base-class bias and novel-class overconfidence.

Quantitative Performance

On the PASCAL-$5^i$ dataset, PCN achieved a 53.81% Harmonic Mean (H-mean) in the 1-shot setting, surpassing the previous SOTA of 44.86%. On the more challenging COCO-$20^i$ dataset, PCN reached an H-mean of 32.48%, representing a nearly 10% jump over previous benchmarks.

Qualitative Insights

Visual analysis (Figure 1) reveals that baseline models often "hallucinate" novel classes over base-class regions (indicated by red circles). PCN's calibration module effectively suppresses these false positives. By separately processing base and novel heatmaps ($y_{Base}$ HM and $y_{Novel}$ HM) and calibrating them, PCN produces cleaner masks that accurately distinguish between similar-looking base and novel objects.

Conclusion

The Prediction Calibration Network (PCN) establishes a new benchmark for GFSS by replacing weight fusion with a more balanced score-fusion and calibration strategy. By leveraging an efficient Transformer-based module, PCN mitigates the inherent biases of few-shot learning. Future research may focus on modeling object-part relationships to further enhance generalization from minimal examples.

Summary updated

Abstract

Generalized Few-shot Semantic Segmentation (GFSS) aims to segment each image pixel into either base classes with abundant training examples or novel classes with only a handful of (e.g., 1-5) training images per class. Compared to the widely studied Few-shot Semantic Segmentation FSS, which is limited to segmenting novel classes only, GFSS is much under-studied despite being more practical. Existing approach to GFSS is based on classifier parameter fusion whereby a newly trained novel class classifier and a pre-trained base class classifier are combined to form a new classifier. As the training data is dominated by base classes, this approach is inevitably biased towards the base classes. In this work, we propose a novel Prediction Calibration Network PCN to address this problem. Instead of fusing the classifier parameters, we fuse the scores produced separately by the base and novel classifiers. To ensure that the fused scores are not biased to either the base or novel classes, a new Transformer-based calibration module is introduced. It is known that the lower-level features are useful of detecting edge information in an input image than higher-level features. Thus, we build a cross-attention module that guides the classifier's final prediction using the fused multi-level features. However, transformers are computationally demanding. Crucially, to make the proposed cross-attention module training tractable at the pixel level, this module is designed based on feature-score cross-covariance and episodically trained to be generalizable at inference time. Extensive experiments on PASCAL-$5^{i}$ and COCO-$20^{i}$ show that our PCN outperforms the state-the-the-art alternatives by large margins.
BibTeX
@article{lu2023prediction,
  author = {Zhihe Lu and Sen He and Da Li and Yi-Zhe Song and Tao Xiang},
  title = {{Prediction Calibration for Generalized Few-shot Semantic Segmentation}},
  journal = {IEEE Trans. Image Process.},
  year = {2023},
  url = {https://openreview.net/forum?id=NAt5PaEGRK},
}
Figure