Abstract
Few-shot segmentation (FSS) aims to adapt a model to novel classes using only a few annotated exemplars. Traditional methods rely on semantic-level prototypes from support images, which often discard critical pixel-wise information. We propose the Cycle-Consistent Transformer (CyCTR) to aggregate dense, pixel-wise support features into query features. To prevent the aggregation of irrelevant or harmful features—a common issue in dense attention—we introduce a novel cycle-consistent attention mechanism. This mechanism acts as a hard filter, ensuring query features only attend to semantically consistent support pixels. Our approach achieves state-of-the-art results on PASCAL-$5^i$ and COCO-$20^i$, outperforming previous methods by significant margins.
Introduction: The Challenge of Feature Confusion
Few-shot segmentation requires models to generalize to unseen categories from minimal data. Historically, the standard approach has been to compress support image features into class-wise "prototypes." While these provide high-level guidance, they lose fine-grained details necessary for precise boundaries.
Recent attempts to use pixel-to-pixel attention face the challenge of feature confusion. Because support and query images vary in scale, pose, and background, a query pixel might mistakenly align with a visually similar but semantically irrelevant pixel in the support image. For example, a background pixel in the query image might match a foreground pixel in the support image due to shared local textures, leading to false positives.
Methodology: Cycle-Consistent Transformer (CyCTR)
To leverage dense features without the risk of noise, we propose the Cycle-Consistent Transformer (CyCTR). CyCTR shifts away from prototype compression, using a transformer architecture to aggregate pixel-wise support features directly into the query representation.
The core innovation is the Cross-Alignment Block with Cycle-Consistent Attention. This block performs cross-attention between query and support features but applies a deterministic cycle-consistency check. For a given support pixel, the model finds its nearest neighbor in the query features, then finds that query pixel's nearest neighbor back in the support features. If the cycle returns to a pixel of the same semantic category (e.g., both are foreground), the connection is maintained; otherwise, the attention weight is forced to zero.
CyCTR also includes a Self-Alignment Block that uses learnable deformable attention to capture global context within the query image. To manage the computational load of pixel-wise attention, especially in multi-shot settings, we employ a mask-guided sparse sampling strategy to select a representative subset of support tokens.
Experiments and Results
We evaluated CyCTR on two standard benchmarks, PASCAL-$5^i$ and COCO-$20^i$. Our method consistently outperformed previous state-of-the-art models like PFENet:
- PASCAL-$5^i$: Achieved 64.0% mIoU for 1-shot and 67.5% mIoU for 5-shot, surpassing PFENet by 3.2% and 5.6% respectively.
- COCO-$20^i$: Achieved 40.3% mIoU for 1-shot and 45.6% mIoU for 5-shot, representing a significant improvement over previous benchmarks.
Ablation studies confirmed that cycle-consistent attention improved performance by 0.6% mIoU. Notably, allowing the model to attend to cycle-consistent background pixels added another 0.5% mIoU, validating our hypothesis that background context is useful when filtered correctly.
Conclusion
The Cycle-Consistent Transformer demonstrates that dense, pixel-wise feature aggregation is superior to prototype-based methods for few-shot segmentation when paired with a robust filtering mechanism. By repurposing cycle-consistency as a forward-pass attention filter, CyCTR effectively manages noise and leverages both foreground and background context to achieve superior segmentation accuracy.
Summary updated