Abstract
Few-shot semantic segmentation (FSS) aims to identify novel object categories using minimal labeled support examples. A persistent challenge in FSS is overfitting: decoders often become overly specialized to training classes, losing the generalizable features of pre-trained backbones. This paper introduces a framework that leverages the inherent semantic grouping abilities of Vision Transformers (ViTs) through Relationship Descriptors (RDs). By preserving the matching logic of pre-trained models, this approach achieves state-of-the-art performance on PASCAL-$5^i$ and COCO-$20^i$ benchmarks, particularly in generalized FSS settings.
The Overfitting Challenge in Few-Shot Segmentation
Standard FSS methods typically train a decoder to map features from a pre-trained backbone to specific class labels. However, when trained on a limited set of "base" classes, these decoders often overfit, failing to generalize to "novel" classes encountered at test time. While Vision Transformers (ViTs) offer powerful global dependency modeling, simply fine-tuning them with standard decoders does not inherently solve this generalization gap.
Vision Transformers as Natural Groupers
A key insight of this work is that pre-trained ViTs (e.g., those trained on ImageNet) already possess a remarkable ability to group semantically similar pixels without any class-specific fine-tuning.
As shown in Figure 4, the ViT backbone produces feature embeddings where related patches (like parts of a person or a bicycle) naturally cluster. This "matching logic" is more robust and class-agnostic than that of traditional CNNs like ResNet50, providing a stronger foundation for few-shot tasks.
Methodology: Relationship Descriptors and Multi-layer Fusion
To harness the ViT's grouping ability without inducing overfitting, the authors propose a framework centered on Relationship Descriptors (RDs).
1. Relationship Descriptors (RDs)
Instead of relying solely on a class prototype (the average feature vector of a support object), the model calculates an RD. This is an element-wise product between query patch embeddings and the class prototype, weighted by cosine similarity. This mechanism encodes how features match, rather than just what they match, effectively turning the decoder into a relationship-matching engine rather than a fixed classifier.
2. Framework Architecture
The overall architecture extracts features from multiple depths of the transformer (specifically the last three layers) to capture both high-level semantics and fine-grained spatial details.
As illustrated in Figure 5, the RD Module generates queries for a transformer-based decoder by concatenating class prototypes with their corresponding RDs. This allows the decoder to attend to both the content of the support image and the spatial relationships within the query image. Additionally, the model employs Momentum Prototype Updates to maintain stable, representative features for base classes during training.
Experimental Results
The model was evaluated on the PASCAL-$5^i$ and COCO-$20^i$ benchmarks in both Binary FSS and Generalized FSS (GFSS) settings.
- State-of-the-Art Performance: In the 1-shot GFSS setting on PASCAL-$5^i$, the model achieved a mean Intersection over Union (mIoU) of 51.0 for novel classes, more than doubling the previous state-of-the-art of 23.2 (CAPL-ViT).
- Robust Generalization: The method consistently outperformed existing techniques even without the test-time fine-tuning often required by competing models.
- Layer Synergy: Ablation studies confirmed that fusing RDs from multiple transformer layers provides a significant performance boost over single-layer implementations.
Conclusion
By shifting the focus from learning class-specific classifiers to preserving the inherent matching logic of pre-trained Vision Transformers, this work provides a robust solution to the overfitting problem in few-shot semantic segmentation. While resolution constraints of ViT patches remain a challenge for segmenting very small objects, the use of Relationship Descriptors represents a significant step forward in building generalizable vision systems.
Summary updated