TL;DR
This paper introduces a novel approach to Referring Image Segmentation (RIS) by effectively bridging the powerful segmentation capabilities of SAM with the rich visual-language understanding of CLIP. Their method designs a new module that extracts and adapts features to guide SAM, enabling accurate segmentation of objects described by natural language, even in more complex scenarios like Generalized Referring Expression Segmentation (GRES).
What Problem Does This Paper Address?
Referring Image Segmentation (RIS) is a challenging task that requires an AI to pinpoint and segment an object in an image based on a natural language description. This goes beyond simple object detection or semantic segmentation because it involves understanding the nuances of language, including attributes and spatial relationships, and then precisely delineating the object's boundaries. While recent advancements like CLIP (for vision-language understanding) and SAM (for zero-shot segmentation) offer powerful individual capabilities, effectively combining them for RIS, especially for more complex scenarios like segmenting multiple objects or when no object is present (GRES), remains an open problem. Existing methods often struggle with the computational cost of large language models or are limited by assumptions of a single target object.
Methods
The core of this work is a proposed model that integrates CLIP and SAM. The key innovation lies in a Dense Feature Extraction Module (DFEM) and a Dense Feature Adaptation Module (DFAM).
- Feature Encoding: The model uses pre-trained CLIP and SAM vision encoders, along with CLIP and BERT language encoders. Crucially, the CLIP and SAM vision encoders are kept fixed during training to leverage their pre-trained strengths.
- DFEM: This module fuses CLIP's visual features with BERT's language features using a Pixel Word Attention Module (PWAM). This fusion aims to create dense features that capture both visual discriminative power and language specificity for the target object.
- DFAM: This module, based on the Mask2Former decoder, acts as a trainable component during training. It takes the fused features from DFEM and SAM's visual features to adapt them into a format suitable for SAM's mask decoder. DFAM is designed to reorganize features and is not required during inference, making the final model efficient.
- Sparse Feature Design: A "sparse feature" is designed to act as a query for the dense features. This sparse feature is derived from CLIP's noun categories (extracted using prompt templates) and BERT's language features, incorporating cross-attention.
- Training Objective: The training incorporates a novel VL alignment objective that treats language expressions from other samples in the mini-batch as negative examples. This objective is applied to both CLIP and BERT features, allowing the model to learn robust correspondences. Importantly, this objective does not assume a single target object, making it suitable for GRES.
Datasets
The paper evaluates its approach on several standard RIS datasets:
- RefCOCO/+/g: These datasets contain natural language expressions and corresponding segmentation masks, varying in expression length and complexity.
- gRefCOCO and Ref-zom: These datasets introduce a more challenging setting called Generalized Referring Expression Segmentation (GRES), where there might be zero, one, or multiple target objects.
Key Insights and Ideas
- Bridging SAM and CLIP is promising: The paper demonstrates that combining the strengths of SAM (segmentation) and CLIP (vision-language understanding) is a viable and effective strategy for RIS.
- Feature adaptation is crucial: Simply feeding CLIP features to SAM isn't enough. The proposed DFAM module is essential for adapting and reorganizing features to effectively guide SAM's mask decoder.
- No single-target assumption is key for GRES: By designing a training objective that doesn't assume a single target object, the model naturally extends to the more complex GRES setting, a significant advantage.
- Fixed encoders are efficient: Leveraging pre-trained, fixed CLIP and SAM encoders significantly reduces the number of trainable parameters, leading to better training efficiency.
Key Results
- The proposed model achieves competitive or superior performance across various datasets (RefCOCO/+/g, gRefCOCO, Ref-zom) compared to state-of-the-art methods like ReLA and CGFormer, particularly in terms of mIoU and oIoU.
- On gRefCOCO, the model demonstrates clear superiority in both RIS and GRES settings.
- On Ref-zom, the model achieves competitive mask estimation and outperforms in non-target prediction accuracy.
- The model shows good training data efficiency, exhibiting smaller performance drops than ReLA when trained with reduced amounts of data.
- Qualitative results show accurate mask predictions, even for small objects, though challenges remain for complex comparative expressions or negations.
Limitations and Future Work
- Computational Cost: While the trainable parameters are reduced, the inference phase still incurs higher computational costs due to the use of CLIP and SAM, representing a trade-off between accuracy and efficiency.
- Mini-batch Contrastive Learning: The paper acknowledges that the inter-mini-batch contrastive learning objective can be challenging when language expressions from different samples are similar but not perfectly matching (e.g., "person with bag" vs. "person with a bag on the left side"). This can lead to incorrect vision-language alignment. Addressing this fundamentally might require more advanced techniques like LLMs for better correspondence establishment.
- Dataset Dependency: The authors note that their method is particularly effective when language expressions are varied and can specifically identify target objects. For simpler datasets like RefCOCO, the performance gain might be less pronounced due to the nature of the VL alignment objective.
- Future Work: The paper suggests that future work could explore more advanced methods for establishing correspondences within the mini-batch, potentially involving LLMs, to further improve the VL alignment objective.
Summary updated