Abstract
Few-shot segmentation (FSS) aims to identify and segment novel object classes using only a limited number of annotated support samples. While recent advancements have shifted from prototype-based methods to cross-attention mechanisms to preserve spatial details, they often suffer from background mismatch and foreground-background entanglement. This paper introduces the Self-Calibrated Cross Attention Network (SCCAN), which allows query pixels to attend to both themselves and aligned support patches. This self-calibration mechanism effectively disentangles foreground and background features, leading to significant performance gains. SCCAN achieves state-of-the-art results on standard benchmarks, including a 5.6% mIoU improvement on the COCO-20$^i$ dataset.
Introduction: The Challenge of Few-Shot Segmentation
Few-shot segmentation is a critical task in computer vision where a model must generalize to unseen classes with minimal supervision. Early approaches relied on global prototypes—feature vectors that represent an entire class. However, compressing an object into a single vector often results in the loss of fine-grained spatial information.
To address this, researchers adopted cross-attention mechanisms, enabling dense pixel-to-pixel matching between query and support images. Despite their potential, standard cross-attention models face two primary hurdles:
- Background (BG) Mismatch: Query background pixels often lack corresponding features in the support foreground. Standard attention forces these pixels to match with the support foreground, introducing noise.
- FG-BG Entanglement: When both query foreground and background pixels attend to the same support features, their resulting representations become entangled, making it difficult for the decoder to produce accurate segmentation masks.
Methodology: Self-Calibrated Cross Attention
The core innovation of this work is the Self-Calibrated Cross Attention (SCCA) block. Unlike traditional cross-attention, which only considers support features, SCCA allows query pixels to look at a combined pool of features from both the query image itself and the support image.
Key Components
- Patch Alignment (PA): To handle spatial variations where objects appear in different locations across images, the PA module splits images into patches. It aligns each query patch with its most visually similar support patch before calculating attention.
- Self-Calibrated Cross Attention: For any query patch, the keys and values are derived from a group containing patches from the query image and the aligned support patches. This allows background pixels to find their "best match" within the query image (self-attention), while foreground pixels can still leverage support information (cross-attention).
- Scaled-Cosine (SC) Mechanism: To ensure the model doesn't over-rely on self-attention, the authors employ a stricter scaled dot-product for self-attention and a more flexible cosine similarity for cross-attention. This structural prior encourages the integration of support sample information.
- Pseudo Mask Aggregation (PMA): This module generates an initial coarse mask by comparing query pixels to both foreground and background support features, effectively suppressing initial noise.
Evaluation and Results
SCCAN was evaluated on the PASCAL-5$^i$ and COCO-20$^i$ benchmarks. The results demonstrate that self-calibration provides a substantial advantage in complex scenarios.
- COCO-20$^i$ (5-shot): SCCAN achieved a mean IoU (mIoU) of 59.9%, outperforming the previous state-of-the-art (DCAMA) by 5.6%.
- PASCAL-5$^i$ (1-shot): The model reached 68.3% mIoU, setting a new benchmark for the dataset.
- Qualitative Analysis: Visual inspections reveal that SCCAN is significantly better at distinguishing target objects from visually similar background elements, such as correctly identifying a horse while ignoring a human rider.
Conclusion and Future Work
The Self-Calibrated Cross Attention Network addresses the fundamental limitations of standard cross-attention in few-shot segmentation. By providing a self-referential path for background pixels, it achieves superior feature disentanglement. While highly effective, the current $k$-shot strategy relies on simple feature averaging, which may dilute spatial information in diverse multi-shot scenarios. Future research will explore more sophisticated fusion strategies to preserve spatial distinctness across multiple support samples.
Summary updated