Abstract
Generalized Few-shot Semantic Segmentation (GFSS) aims to segment each image pixel into either base classes with abundant training examples or novel classes with only a handful of training images per class. Compared to the widely studied Few-shot Semantic Segmentation (FSS), which is limited to segmenting novel classes only, GFSS is much under-studied despite being more practical. Existing approaches to GFSS are based on classifier parameter fusion whereby a newly trained novel class classifier and a pre-trained base class classifier are combined. As the training data is dominated by base classes, this approach is inevitably biased towards the base classes. In this work, we propose a novel Prediction Calibration Network (PCN) to address this problem. Instead of fusing the classifier parameters, we fuse the scores produced separately by the base and novel classifiers. To ensure that the fused scores are not biased, a new Transformer-based calibration module is introduced. This module uses feature-score cross-covariance and is episodically trained to be generalizable. Extensive experiments on PASCAL-$5^{i}$ and COCO-$20^{i}$ show that our PCN outperforms the state-of-the-art alternatives by large margins.
Introduction
In realistic computer vision applications, new objects (novel classes) typically appear alongside well-known objects (base classes). While standard Few-shot Semantic Segmentation (FSS) focuses on identifying novel objects using only 1–5 examples, it ignores the base classes during inference. Generalized Few-shot Semantic Segmentation (GFSS) addresses this by requiring the model to segment both base and novel classes simultaneously.
The primary challenge in GFSS is the extreme data imbalance. Base classes have thousands of training samples, while novel classes have very few. Traditional methods that merge classifier weights often result in a heavy bias toward base classes. Conversely, attempts to boost novel class detection can lead to "overconfidence," where base-class pixels are incorrectly labeled as novel ones.
Methodology: Prediction Calibration Network (PCN)
The proposed PCN architecture addresses these challenges through a two-stage fusion and calibration process.
Normalized Score Fusion (NSF)
Instead of merging internal classifier parameters, PCN runs base and novel classifiers separately. It applies a softmax operation to each set of scores independently to normalize them before fusion. This approach significantly reduces the initial bias toward base classes that occurs when weights are merged directly.
Transformer-based Calibration
To resolve the "overconfidence trap" where novel classes misidentify base-class pixels, PCN introduces a Transformer-based calibration module.
- Cross-covariance Attention: Standard Transformers are computationally expensive for pixel-level tasks. PCN treats class scores and feature channels as tokens, computing the covariance between them to maintain efficiency while capturing global context.
- Multi-level Feature Guidance: The module utilizes both high-level semantic features and low-level edge information to ensure that final segmentation masks respect object boundaries and spatial details.
Results and Discussion
PCN was evaluated on the PASCAL-$5^i$ and COCO-$20^i$ benchmarks, demonstrating significant improvements over existing state-of-the-art (SOTA) methods.
Quantitative Performance
On the PASCAL-$5^i$ dataset, PCN achieved a 53.81% Harmonic Mean (H-mean) in the 1-shot setting, surpassing the previous SOTA of 44.86%. On the more challenging COCO-$20^i$ dataset, PCN reached an H-mean of 32.48%, representing a nearly 10% jump over previous benchmarks.
Qualitative Insights
Visual analysis (Figure 1) reveals that baseline models often "hallucinate" novel classes over base-class regions (indicated by red circles). PCN's calibration module effectively suppresses these false positives. By separately processing base and novel heatmaps ($y_{Base}$ HM and $y_{Novel}$ HM) and calibrating them, PCN produces cleaner masks that accurately distinguish between similar-looking base and novel objects.
Conclusion
The Prediction Calibration Network (PCN) establishes a new benchmark for GFSS by replacing weight fusion with a more balanced score-fusion and calibration strategy. By leveraging an efficient Transformer-based module, PCN mitigates the inherent biases of few-shot learning. Future research may focus on modeling object-part relationships to further enhance generalization from minimal examples.
Summary updated