Few-Shot Segmentation via Cycle-Consistent Transformer

Gengwei Zhang, Guoliang Kang, Yunchao Wei, Yi Yang

Abstract

Few-shot segmentation (FSS) aims to adapt a model to novel classes using only a few annotated exemplars. Traditional methods rely on semantic-level prototypes from support images, which often discard critical pixel-wise information. We propose the Cycle-Consistent Transformer (CyCTR) to aggregate dense, pixel-wise support features into query features. To prevent the aggregation of irrelevant or harmful features—a common issue in dense attention—we introduce a novel cycle-consistent attention mechanism. This mechanism acts as a hard filter, ensuring query features only attend to semantically consistent support pixels. Our approach achieves state-of-the-art results on PASCAL-$5^i$ and COCO-$20^i$, outperforming previous methods by significant margins.

Introduction: The Challenge of Feature Confusion

Few-shot segmentation requires models to generalize to unseen categories from minimal data. Historically, the standard approach has been to compress support image features into class-wise "prototypes." While these provide high-level guidance, they lose fine-grained details necessary for precise boundaries.

Recent attempts to use pixel-to-pixel attention face the challenge of feature confusion. Because support and query images vary in scale, pose, and background, a query pixel might mistakenly align with a visually similar but semantically irrelevant pixel in the support image. For example, a background pixel in the query image might match a foreground pixel in the support image due to shared local textures, leading to false positives.

Feature confusion in FSS. A foreground pixel $p_1$ in the support image may incorrectly align with a background pixel $p_2$ in the query image due to visual similarity, while $p_3$ represents a correct background match.

Methodology: Cycle-Consistent Transformer (CyCTR)

To leverage dense features without the risk of noise, we propose the Cycle-Consistent Transformer (CyCTR). CyCTR shifts away from prototype compression, using a transformer architecture to aggregate pixel-wise support features directly into the query representation.

The core innovation is the Cross-Alignment Block with Cycle-Consistent Attention. This block performs cross-attention between query and support features but applies a deterministic cycle-consistency check. For a given support pixel, the model finds its nearest neighbor in the query features, then finds that query pixel's nearest neighbor back in the support features. If the cycle returns to a pixel of the same semantic category (e.g., both are foreground), the connection is maintained; otherwise, the attention weight is forced to zero.

Comparison of FSS architectures: (a) Prototype-based, (b) Clustering-based, (c) Graph-Attention, and (d) the proposed Cycle-Consistent Transformer (CyCTR) which utilizes dense features with a cycle-consistency filter.

CyCTR also includes a Self-Alignment Block that uses learnable deformable attention to capture global context within the query image. To manage the computational load of pixel-wise attention, especially in multi-shot settings, we employ a mask-guided sparse sampling strategy to select a representative subset of support tokens.

Experiments and Results

We evaluated CyCTR on two standard benchmarks, PASCAL-$5^i$ and COCO-$20^i$. Our method consistently outperformed previous state-of-the-art models like PFENet:

  • PASCAL-$5^i$: Achieved 64.0% mIoU for 1-shot and 67.5% mIoU for 5-shot, surpassing PFENet by 3.2% and 5.6% respectively.
  • COCO-$20^i$: Achieved 40.3% mIoU for 1-shot and 45.6% mIoU for 5-shot, representing a significant improvement over previous benchmarks.

Ablation studies confirmed that cycle-consistent attention improved performance by 0.6% mIoU. Notably, allowing the model to attend to cycle-consistent background pixels added another 0.5% mIoU, validating our hypothesis that background context is useful when filtered correctly.

Conclusion

The Cycle-Consistent Transformer demonstrates that dense, pixel-wise feature aggregation is superior to prototype-based methods for few-shot segmentation when paired with a robust filtering mechanism. By repurposing cycle-consistency as a forward-pass attention filter, CyCTR effectively manages noise and leverages both foreground and background context to achieve superior segmentation accuracy.

Summary updated

Abstract

Few-shot segmentation aims to train a segmentation model that can fast adapt to novel classes with few exemplars. The conventional training paradigm is to learn to make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as conditional information. These methods cannot utilize all pixel-wise support information for the query predictions, which is however critical for the segmentation task. In this paper, we focus on utilizing pixel-wise relationships between support and query images to facilitate the few-shot segmentation task. We design a novel Cycle-Consistent TRansformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using a novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared to previous state-of-the-art methods. Specifically, on Pascal-$5^i$ and COCO-$20^i$ datasets, we achieve 67.5% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-art methods by 5.6% and 7.1% respectively.
BibTeX
@inproceedings{zhang2021few,
  author = {Gengwei Zhang and Guoliang Kang and Yunchao Wei and Yi Yang},
  title = {{Few-Shot Segmentation via Cycle-Consistent Transformer}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=LWH-C1HoQG_},
}
Figure