Few-Shot Segmentation via Cycle-Consistent Transformer

Gengwei Zhang, Guoliang Kang, Yi Yang, Yunchao Wei

TL;DR

CyCTR (Cycle-Consistent TRansformer) moves beyond the "prototype" bottleneck in few-shot segmentation by using pixel-wise attention to align support and query images. By introducing a novel cycle-consistency mechanism, the model dynamically filters out "harmful" support features that don't match the query's context, leading to a massive +7.1% mIoU jump over previous state-of-the-art methods on the COCO dataset.

What Problem Does This Paper Address?

Few-shot segmentation (FSS) is the task of segmenting a novel object in a "query" image given only a few labeled "support" examples.

The traditional approach involves creating a prototype—a single vector representing the average features of the support object. While simple, this "compresses" away critical spatial details, fine-grained textures, and boundaries. Newer methods attempted to use pixel-to-pixel attention to preserve this spatial information. However, they ran into a "noise" problem: if the support image has a different background or the object appears in a vastly different scale or color, the attention mechanism gets confused. It tries to aggregate information from support pixels that have nothing to do with the query object, leading to "biased" features and poor segmentation.

Pixel-wise relationships between support and query images

Methods

To overcome these limitations, the authors propose the Cycle-Consistent TRansformer (CyCTR). This architecture consists of a series of encoders designed to aggregate pixel-level information without falling prey to noisy, irrelevant features.

Comparison of few-shot segmentation architectures, highlighting the proposed CyCTR (d)

Each encoder features two main components:

  1. Self-Alignment Block: This block focuses on the query image itself. It uses a "PredAtten" (Predicted Attention) mechanism—inspired by deformable transformers—to help each query pixel gather global context from other relevant pixels within the same image.
  2. Cross-Alignment Block (The Core Innovation): This block allows query pixels to "look" at support pixels. To prevent the model from being distracted by irrelevant support features, the authors introduce Cycle-Consistent Attention:
  3. For a support pixel $j$, the model finds its most similar query pixel $i^*$.
  4. Then, for that query pixel $i^*$, it finds its most similar support pixel $j^*$.
  5. If the original support pixel $j$ and the "cycled" support pixel $j^*$ do not share the same class label (based on the support mask), pixel $j$ is considered "inconsistent" and is masked out of the attention calculation.

To keep the transformer computationally efficient, the method employs mask-guided sparse sampling. This selects a fixed number of support tokens (e.g., 600) from both the foreground and background, ensuring the model sees a balanced view of the support image without processing every single pixel.

Datasets

The model was evaluated on two standard few-shot segmentation benchmarks:

  • Pascal-$5^i$: A subset of PASCAL VOC 2012 with additional SBD annotations, divided into four splits.
  • COCO-$20^i$: A more challenging and large-scale benchmark derived from MS COCO, also divided into four splits.

Key Insights and Ideas

  • Prototypes are Bottlenecks: Aggregating a whole object into one vector loses the fine-grained texture and boundary information needed for dense prediction. Pixel-wise alignment is superior but requires a robust "filter."
  • Background Matters: Unlike many previous methods that only look at the support foreground, CyCTR samples from the support background too. This provides contrastive information that helps the model better distinguish the object from its surroundings in the query image.
  • Cycle-Consistency as a Denoising Filter: The most elegant idea here is using cycle-consistency. It acts as a non-learnable, plug-and-play filter that ensures the query image only "trusts" support pixels that have a clear, reciprocal relationship with the query's content.
  • Dynamic Adaptation: Because the cycle-consistency check happens during the forward pass, the model effectively "re-weights" the importance of support pixels for every unique query-support pair.

Key Results

CyCTR achieved significant performance gains over previous state-of-the-art (SOTA) methods:

  • Pascal-$5^i$: Achieved 66.6% mIoU in the 5-shot setting, outperforming the previous SOTA by 4.6%.
  • COCO-$20^i$: Achieved 45.6% mIoU in the 5-shot setting, outperforming the previous SOTA by 7.1%.
  • Ablation Studies: The researchers showed that adding the cross-alignment block improved mIoU, but adding the cycle-consistency to that block provided an additional significant boost, proving that filtering "harmful" support features is vital.

Limitations and Future Work

  • Over-suppression: The authors noted that in some cases, the cycle-consistency mechanism might be too strict, potentially suppressing valid positive regions if the visual difference between support and query is extreme (e.g., a "cattle" in the support vs. a "milk cow" in the query).
  • Computational Overhead: While sparse sampling helps, transformers are still more computationally expensive than simple prototype-based linear layers.
  • Future Directions: Investigating ways to make the cycle-consistency more robust to extreme intra-class variations is a suggested path forward.

Summary updated

Abstract

Few-shot segmentation aims to train a segmentation model that can fast adapt to novel classes with few exemplars. The conventional training paradigm is to learn to make predictions on query images conditioned on the features from support images. Previous methods only utilized the semantic-level prototypes of support images as the conditional information. These methods cannot utilize all pixel-wise support information for the query predictions, which is however critical for the segmentation task. In this paper, we focus on utilizing pixel-wise relationships between support and target images to facilitate the few-shot semantic segmentation task. We design a novel Cycle-Consistent Transformer (CyCTR) module to aggregate pixel-wise support features into query ones. CyCTR performs cross-attention between features from different images, i.e. support and query images. We observe that there may exist unexpected irrelevant pixel-level support features. Directly performing cross-attention may aggregate these features from support to query and bias the query features. Thus, we propose using a novel cycle-consistent attention mechanism to filter out possible harmful support features and encourage query features to attend to the most informative pixels from support images. Experiments on all few-shot segmentation benchmarks demonstrate that our proposed CyCTR leads to remarkable improvement compared to previous state-of-the-art methods. Specifically, on Pascal-5^i and COCO-20^i datasets, we achieve 66.6% and 45.6% mIoU for 5-shot segmentation, outperforming previous state-of-the-art by 4.6% and 7.1% respectively.
BibTeX
@inproceedings{zhang2021few,
  author = {Gengwei Zhang and Guoliang Kang and Yi Yang and Yunchao Wei},
  title = {{Few-Shot Segmentation via Cycle-Consistent Transformer}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=LWH-C1HoQG_},
}
Figure