TL;DR
This paper introduces RiFeNet, a novel network for few-shot semantic segmentation that tackles the challenges of semantic ambiguity within classes and similarity between classes. It achieves this by incorporating an unlabeled branch for robust feature learning and a multi-level prototype system for better foreground-background distinction, outperforming existing methods on PASCAL-5i and COCO benchmarks.
What Problem Does This Paper Address?
Semantic segmentation, the task of assigning a class label to every pixel in an image, is crucial for many applications. However, acquiring pixel-level annotations is labor-intensive and expensive. Few-shot semantic segmentation aims to address this by training models with very limited labeled data (support set) and then applying them to new images (query set). The core challenge in few-shot semantic segmentation lies in extracting reliable class-specific features from scarce examples. Existing methods struggle with two main issues:
- Intra-class semantic ambiguity: Different instances of the same class can have significant variations in appearance (shape, pose, texture). This makes it hard for models to learn consistent representations, leading to errors when segmenting new instances.
- Inter-class similarity: Objects from different classes can look very similar, making it difficult to distinguish between foreground and background, especially when they have comparable textures or patterns.
Methods
RiFeNet proposes a novel architecture to tackle these challenges. It builds upon the standard two-branch (support and query) structure but introduces two key innovations:
- Unlabeled Branch for Semantic Consistency: To improve the model's ability to learn features robust to intra-class variations, RiFeNet incorporates an additional "unlabeled branch" during training. This branch uses unlabeled data (resampled from the training set) and is trained with a self-supervised objective (Dice loss). The idea is to teach the model to extract intrinsic features that are consistent across different appearances of the same class, acting as a regularization mechanism. Crucially, this branch is discarded during inference, meaning no extra unlabeled data or computation is needed at test time.
- Multi-Level Prototype Generation and Interaction: To enhance the distinction between foreground and background, RiFeNet introduces a novel module for generating and interacting with prototypes at multiple levels of granularity.
- Global Prototypes: Extracted from the support set, these capture high-level semantic abstractions of the entire category.
- Local Prototypes: Extracted from the query set, these capture fine-grained appearance details. These global and local prototypes are then interacted with and combined with query features. This multi-grained approach allows for a richer representation that better discriminates between similar categories and improves the foreground-background separation.
The overall architecture consists of three main blocks: multi-level prototype generation, multi-level prototype interaction, and feature activation (using a transformer encoder).
Datasets
The paper evaluates RiFeNet on two standard benchmarks for few-shot semantic segmentation:
- PASCAL-5²: A dataset derived from PASCAL VOC, specifically designed for few-shot segmentation tasks.
- COCO: The Common Objects in Context dataset, also adapted for few-shot segmentation evaluation.
The unlabeled branch uses data resampled from the training splits of these datasets.
Key Insights and Ideas
- Leveraging Unlabeled Data Without Inference Overhead: The unlabeled branch is a clever way to improve feature robustness by using readily available unlabeled data during training. Its removal during inference is a significant advantage, as it doesn't increase computational cost or complexity for deployment.
- Multi-Level Prototypes for Fine-Grained Discrimination: The combination of global (abstract) and local (detailed) prototypes is key to overcoming inter-class similarity. This hierarchical approach provides complementary information that helps distinguish between visually similar objects.
- Self-Supervision for Feature Consistency: The self-supervised loss on the unlabeled branch effectively guides the model to learn features that are invariant to appearance variations within a class, directly addressing the semantic ambiguity problem.
- The Importance of "Guidance" in Unlabeled Branches: Ablation studies show that an unlabeled branch without guidance from query prototypes can actually hurt performance, highlighting the necessity of integrating information from both support and query branches.
Key Results
RiFeNet demonstrates state-of-the-art performance on both PASCAL-5² and COCO datasets.
- PASCAL-5²: RiFeNet achieves a mean IoU (mIoU) of 69.6% in the 1-shot setting and 69.2% in the 5-shot setting (using ResNet101 backbone), outperforming previous methods like DCAMA by a significant margin (e.g., 2.7% in 1-shot).
- COCO: RiFeNet achieves an mIoU of 48.6% in the 1-shot setting (using ResNet50 backbone), surpassing the previous best method DCAMA by 0.8%.
Qualitative results show RiFeNet's ability to maintain foreground semantic consistency even with significant intra-class variations and to better distinguish between similar foreground and background objects.
Limitations and Future Work
- Impact of Unlabeled Data Ratio: The paper notes that the effectiveness of the unlabeled branch decreases as the ratio of unlabeled to labeled images decreases (e.g., in the 5-shot setting compared to 1-shot). This suggests that while beneficial, the impact might be more pronounced when the labeled data is extremely scarce.
- Optimal Number of Unlabeled Images: Ablation studies indicate that there's an optimal number of unlabeled images (found to be 2 in their experiments) for the meta-training process. Too few might not provide enough regularization, while too many could potentially distract the model from the primary task.
- Future Work: The authors suggest exploring different design choices for multi-level prototypes and unlabeled branches, as well as investigating the impact of various hyper-parameters, which are detailed in the appendix. The success of RiFeNet also opens avenues for applying its core ideas to other few-shot learning tasks.
Summary updated