Abstract
Few-shot semantic segmentation (FSS) aims to segment novel object categories using minimal annotated examples. However, existing models often struggle with background noise and intra-class variability. This paper introduces a Transformer-based network that identifies and mitigates three specific architectural biases: inherent, attention, and spatial-aware. By incorporating a Target Enhancement Module (TEM), a Dual Constraint Aggregation Module (DCAM), and a Dual Classification Module (DCM), the proposed method achieves state-of-the-art performance on standard benchmarks while maintaining high parameter efficiency.
Introduction: The Challenge of Background Noise
Few-Shot Semantic Segmentation (FSS) requires models to generalize to unseen categories from just a few support images. A primary hurdle is "background noise," where models are distracted by clutter or previously seen classes.
The authors identify three core biases hindering current FSS performance:
- Inherent Bias: Pre-trained backbones favor common objects from their training set (e.g., humans) over specific target objects (e.g., a bottle).
- Attention Bias: High intra-class variability makes it difficult for standard attention mechanisms to match support and query features.
- Spatial-aware Bias: Over-reliance on semantic relevance at the expense of spatial consistency leads to fragmented segmentation masks.
Methodology
The proposed architecture addresses these biases through three specialized modules:
Target Enhancement Module (TEM)
To counter inherent bias, the TEM utilizes a Multi-Scale Local-Aware Modulation Transformer. It replaces standard feed-forward layers with an Invertible Neural Network (INN). This design choice is crucial for preserving fine-grained spatial details that are typically lost during downsampling, ensuring the model retains information necessary for precise segmentation.
Dual Constraint Aggregation Module (DCAM)
DCAM addresses attention bias by mimicking human visual search. It employs a two-layer constraint system that identifies high-confidence anchor points in the query image to guide feature retrieval. Learnable "intra-class difference" vectors are also used to bridge the gap between varying appearances of the same object class.
Dual Classification Module (DCM)
To resolve spatial-aware bias, the DCM decouples the task into semantic alignment (category identification) and spatial alignment (localization). By processing these separately and then merging them, the model produces more coherent and spatially accurate masks.
Experimental Results
The model was evaluated on the PASCAL-5$^i$ and COCO-20$^i$ datasets using VGG-16 and ResNet-50 backbones.
Quantitative Performance
The proposed method set new benchmarks:
- PASCAL-5$^i$: 67.0% mIoU (1-shot) and 70.5% mIoU (5-shot) with ResNet-50.
- COCO-20$^i$: 47.5% mIoU (1-shot) and 54.7% mIoU (5-shot).
Notably, the ResNet-50 version achieves these results with only 1.1M learnable parameters, making it significantly more efficient than competing models that often require 15M to 40M parameters.
Qualitative Analysis
Visualizations demonstrate the model's robustness against distractors. As shown in the heatmaps, the model can distinguish the target object even when "distractor" objects (like a person) are prominent in the scene.
Conclusion and Future Work
By systematically addressing architectural biases, the Transformer-Based Adaptive Prototype Matching Network provides a lean and powerful solution for FSS. While highly effective, future research could explore optimizing inference latency for the multi-scale Transformer blocks and improving robustness in extremely cluttered environments where foreground distributions might be misleading.
Summary updated