Graph-guided Cross-composition Feature Disentanglement for Compositional Zero-shot Learning

Yuxia Geng, Runkai Zhu, Jiaoyan Chen, Jintai Chen, Xiang Chen, Zhuo Chen, Shuofei Qiao, Yuxiang Wang, Xiaoliang Xu, Sheng-Jun Huang

TL;DR

This paper introduces DCDA, a novel framework that enhances CLIP for Compositional Zero-Shot Learning (CZSL) by disentangling visual features of attributes and objects. DCDA uses graph-guided adapters to ensure these primitive features remain general across diverse compositions, significantly improving performance on CZSL benchmarks.

What Problem Does This Paper Address?

Compositional Zero-Shot Learning (CZSL) aims to recognize novel combinations of attributes and objects (e.g., "red apple") by learning from seen combinations (e.g., "red tomato," "green apple"). A key challenge is that the visual representation of an attribute (like "red") can change drastically depending on the object it's paired with (e.g., "red wine" vs. "red tomato"). This "feature divergence" makes it difficult to learn disentangled, generalizable primitive features that work across many different compositions. Existing methods often struggle with this entanglement, leading to poor performance, especially when encountering unseen combinations.

Methods

DCDA tackles this problem by proposing "cross-composition feature disentanglement." The core idea is to leverage multiple related compositions to help disentangle primitive features. This is achieved through a novel architecture built upon the pre-trained vision-language model CLIP.

The key components are:

  • Compositional Graph: A tripartite graph is constructed to represent relationships between attributes, objects, and their compositions. This graph guides the learning process.
  • L-Adapter (Textual): This adapter uses the compositional graph and a Graph Neural Network (GNN) to propagate and aggregate textual features. It helps consolidate attribute and object semantics from various contexts, making textual primitive representations more robust.
  • V-Adapter (Visual): This adapter employs cross-attention between images of primitive-sharing compositions (e.g., "red tomato" and "red wine"). It also uses a "primitive relevance-guided sampling" strategy to select the most informative auxiliary images, helping to extract invariant visual patterns and mitigate feature entanglement.
  • Dual Adapters in CLIP: Both L-Adapters and V-Adapters are inserted into CLIP's frozen text and image encoders, respectively. This allows the model to inject task-specific knowledge without disrupting CLIP's powerful pre-trained representations, thus avoiding overfitting.

Datasets

The paper evaluates DCDA on three standard CZSL benchmarks:

  • MIT-States: A widely used dataset for CZSL.
  • UT-Zappos: A dataset focused on shoe attributes and materials.
  • C-GQA: A more extensive dataset derived from Stanford GQA, known for its low-resolution images.

The experiments are conducted in both "closed world" (where unseen compositions are a subset of all possible combinations) and "open world" (where the model must generalize to all possible unseen combinations) settings.

Key Insights and Ideas

  • Primitive Diversity is Key: The paper highlights that the visual manifestation of a primitive (attribute or object) is highly dependent on its context (other primitives it's paired with). This diversity is a major hurdle for CZSL.
  • Cross-Composition Supervision: The core innovation is using "primitive-sharing" compositions as a form of supervision. By looking at how an attribute appears in multiple different object contexts (and vice-versa), the model can learn more generalizable representations.
  • Adapters as Targeted Enhancements: Inserting specialized adapters (L-Adapter and V-Adapter) into CLIP's frozen encoders is an effective way to adapt a powerful pre-trained model for a specific task like CZSL without extensive fine-tuning.
  • Textual Primitives are Less Entangled: The study suggests that textual primitive features are inherently less entangled than visual ones, making them easier to disentangle. This is reflected in the performance drop when V-Adapters are removed versus when L-Adapters are removed.

Key Results

  • State-of-the-Art Performance: DCDA significantly outperforms previous methods on the MIT-States and UT-Zappos benchmarks, achieving notable improvements in AUC and harmonic mean, and substantial gains in seen accuracy.
  • Effective Disentanglement: t-SNE visualizations show that DCDA's learned attribute and object embeddings are more tightly clustered and discriminative compared to a strong baseline (CAILA). This indicates successful disentanglement.
  • Adapter Contributions: Ablation studies confirm that both L-Adapters and V-Adapters are crucial and complementary. Removing either adapter leads to a performance drop.
  • Sampling Strategy Matters: The proposed primitive-relevance guided (PRG) sampling strategy for auxiliary images in the V-Adapter significantly improves performance over random sampling (RD), especially on datasets with imbalanced primitive distributions.
  • C-GQA Adaptation: For the challenging C-GQA dataset with low-quality images, DCDA demonstrates competitive performance by employing more trainable parameters in the vision encoder.

Limitations and Future Work

  • Computational Overhead: Constructing and updating the compositional graph can become computationally expensive as the number of attributes and objects grows, posing a challenge for real-time applications.
  • Long-Tailed Distributions: While the PRG sampling helps with data imbalance, extremely long-tailed distributions of primitives might still lead to suboptimal disentanglement due to insufficient cross-composition supervision.
  • Future Directions: The authors suggest applying DCDA to other vision-language tasks and developing dynamic graph construction mechanisms for open-vocabulary primitive discovery.

Summary updated

Abstract

Disentanglement of visual features of primitives (i.e., attributes and objects) has shown exceptional results in Compositional Zero-shot Learning (CZSL). However, due to the feature divergence of an attribute (resp. object) when combined with different objects (resp. attributes), it is challenging to learn disentangled primitive features that are general across different compositions. To this end, we propose the solution of cross-composition feature disentanglement, which takes multiple primitive-sharing compositions as inputs and constrains the disentangled primitive features to be general across these compositions. More specifically, we leverage a compositional graph to define the overall primitive-sharing relationships between compositions, and build a task-specific architecture upon the recently successful large pre-trained vision-language model (VLM) CLIP, with dual cross-composition disentangling adapters (called L-Adapter and V-Adapter) inserted into CLIP’s frozen text and image encoders, respectively. Evaluation on three popular CZSL benchmarks shows that our proposed solution significantly improves the performance of CZSL, and its components have been verified by solid ablation studies. Our code and data are available at: https://github.com/zhurunkai/DCDA.
BibTeX
@inproceedings{geng2025graph,
  author = {Yuxia Geng and Runkai Zhu and Jiaoyan Chen and Jintai Chen and Xiang Chen and Zhuo Chen and Shuofei Qiao and Yuxiang Wang and Xiaoliang Xu and Sheng-Jun Huang},
  title = {{Graph-guided Cross-composition Feature Disentanglement for Compositional Zero-shot Learning}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure