TL;DR
This paper introduces Decomposed Fusion with Soft Prompt (DFSP), a novel framework for Compositional Zero-Shot Learning (CZSL) that leverages vision-language models. DFSP effectively handles unseen combinations of states and objects by decomposing language features and using a soft prompt to create a joint representation, significantly outperforming existing methods.
What Problem Does This Paper Address?
Compositional Zero-Shot Learning (CZSL) aims to recognize novel concepts formed by combining known "states" (e.g., "green") and "objects" (e.g., "tiger"). The challenge lies in generalizing to combinations that were not seen during training. Existing approaches often struggle with this generalization: some learn a single representation for the combined state-object, which can be too rigid, while others use separate classifiers for states and objects, neglecting their inherent relationship. This leads to a "domain gap" between seen and unseen compositions, hindering performance.
Methods
The proposed framework, Decomposed Fusion with Soft Prompt (DFSP), tackles this by integrating vision-language models (VLMs) and introducing two key modules:
- Soft Prompt Module (SPM): Instead of using discrete text prompts, DFSP employs a fully learnable "soft prompt" that combines a prefix, state, and object. This learnable prompt is designed to capture a more robust joint representation of states and objects, making it more adaptable to new tasks. The SPM uses a pre-trained CLIP model as its backbone, with frozen image and text encoders.
- Decomposed Fusion Module (DFM): This is the core innovation. Unlike previous methods that might decompose image features, DFSP decomposes language features into separate state and object representations. This is motivated by the observation that language features are less entangled than image features for states and objects. These decomposed language features are then fused with image features through a cross-modal fusion process. This allows the image features to learn distinct relationships with both the state and object, improving their expressiveness and narrowing the domain gap between seen and unseen compositions. The DFM also reduces computational complexity by decomposing the problem from O(n*m) to O(n+m) for compositions.
DFSP supports three fusion strategies: BiF (bidirectional fusion), i2t (image-to-text fusion), and t2i (text-to-image fusion).
Datasets
The paper evaluates DFSP on three widely used benchmark datasets for CZSL:
- MIT-States: Contains natural images with states and objects.
- UT-Zappos: Focuses on images of shoes with various states and objects.
- C-GQA: A large-scale dataset for compositional question answering, also used for CZSL.
The experiments are conducted in both closed-world (testing on unseen compositions from a predefined set) and open-world (testing on any possible composition) settings.
Key Insights and Ideas
- Language Features are More Decomposable: The core insight is that language representations of states and objects are less entangled than their visual counterparts. Decomposing language features allows for more precise separation and better utilization of state and object information.
- Soft Prompts for Better Generalization: Using learnable soft prompts, rather than fixed text prompts, allows the model to adapt more effectively to the specific nuances of the CZSL task and unseen compositions.
- Cross-Modal Fusion with Decomposition: Fusing decomposed language features with image features enables a more nuanced interaction, allowing image features to learn specific relationships with individual states and objects, thereby improving sensitivity to unseen combinations.
- Bridging the Seen-Unseen Domain Gap: The proposed decomposition and fusion strategy directly addresses the domain gap, a critical bottleneck in CZSL.
Key Results
DFSP demonstrates significant improvements over state-of-the-art methods across all three datasets and both closed-world and open-world settings.
- Closed-World: DFSP achieved new state-of-the-art results on MIT-States, UT-Zappos, and CGQA, with notable improvements in AUC (e.g., outperforming CSP by 4.3% on MIT-States) and harmonic mean.
- Open-World: DFSP also achieved the best results in open-world settings, with significant gains in unseen accuracy (e.g., 15.9% on UT-Zappos), highlighting its ability to generalize to novel compositions.
- Ablation Studies: The results confirm the effectiveness of both the Soft Prompt Module (SPM) and the Decomposed Fusion Module (DFM). The DFM, in particular, led to substantial improvements, underscoring the benefits of decomposition and fusion. The t2i fusion method generally performed best.
- Qualitative Results: Visual examples show DFSP's strong ability to predict unseen compositions accurately, even in failure cases where the prediction might be incorrect, it still adheres to the logical combination of state and object.
Limitations and Future Work
- Limitations: The paper does not explicitly detail limitations beyond the inherent challenges of CZSL and the need for effective fusion strategies. The performance might still be sensitive to the quality of the pre-trained VLM.
- Future Work: The authors suggest that the framework's effectiveness on challenging datasets and its strong performance in both closed-world and open-world settings indicate promising avenues for future research in compositional zero-shot learning. The success of the t2i fusion method might also suggest further exploration into text-to-image interactions for CZSL.
Summary updated