Leveraging Sub-class Discimination for Compositional Zero-Shot Learning

Xiaoming Hu, Zilei WangAAAI 2023 · OpenReview

TL;DR

This paper tackles the challenge of Compositional Zero-Shot Learning (CZSL) by introducing a novel approach that leverages "sub-class discrimination." By explicitly accounting for how the visual appearance of attributes and objects changes when combined, the authors propose a method that improves both feature-level representations and classifier prototypes, leading to state-of-the-art performance on benchmark datasets.

What Problem Does This Paper Address?

Compositional Zero-Shot Learning (CZSL) aims to recognize novel combinations of attributes and objects that the model hasn't seen during training. For example, if a model has seen "red car" and "blue car," it should be able to recognize a "red truck" even if it has never encountered that specific combination. A key challenge in CZSL is that the visual appearance of an attribute (like "colorful") can change depending on the object it's paired with. "Colorful car" might involve bright, metallic paints, while "colorful building" could mean a facade with various colored bricks or windows. Previous CZSL methods often overlook this crucial interaction, treating attributes and objects in isolation or relying on static word embeddings, which fail to capture these nuanced visual variations. This paper addresses this gap by proposing a method that explicitly models and leverages these "sub-class" variations.

Methods

The proposed approach, termed Sub-class Discrimination CZSL (SCD-CZSL), operates on two main fronts:

  1. Embedding Alignment at Feature Level: The core idea here is to align the features learned by the "disentangled" recognition branches (one for attributes, one for objects) with features that capture the "composed" understanding of the attribute-object pair. This is achieved by:
  2. Virtual Encodings: Using decoders (Da, Do) to generate virtual attribute and object encodings from a composite embedding. These virtual encodings are expected to contain more sub-class specific information.
  3. Contrastive Learning: Employing contrastive learning to pull the disentangled attribute and object embeddings closer to their corresponding virtual encodings (positive anchors) while pushing them away from other samples (negative samples). This encourages the feature representations to capture sub-class variations.
  4. Auxiliary Composition Encoder: An additional encoder (Ep) generates a composite embedding from the disentangled ones. This composite embedding is then used to further align the disentangled features, ensuring that the sub-class variations are preserved.
  5. Prototype Modulation at Classifier Level: After the features are refined, the classifier prototypes (which represent the "ideal" visual features for each attribute or object) are adjusted to better reflect the sub-class information.
  6. Prototype Modulators (Ma, Mo): These modulators take the composite embedding and use it to dynamically adjust the attribute and object prototypes. This ensures that the classifier can distinguish between different sub-classes of the same primitive concept (e.g., "colorful car" vs. "colorful building").

The overall training loss combines the baseline classification losses with an alignment loss and a modulated classification loss. During inference, only the disentangled CZSL recognition branch is used.

Datasets

The authors evaluate their method on two benchmark CZSL datasets:

  • UT-Zappos: A medium-sized dataset of shoe images, featuring 16 attribute categories and 12 object categories.
  • C-GQA: A larger dataset with 413 attribute categories and 674 object categories.

They note that while MIT-States is a common CZSL dataset, it suffers from significant labeling errors, leading them to opt for UT-Zappos and C-GQA.

Key Insights and Ideas

  • The "Colorful" Example: The paper effectively illustrates the core problem with the "colorful" attribute, showing how its visual manifestation differs between a "colorful car" (orange/blue paint) and a "colorful building" (grey/brown bricks). This highlights the inadequacy of static representations.
  • Sub-class Discrimination is Key: The central thesis is that explicitly modeling and discriminating between sub-classes (e.g., different visual instances of "colorful") is crucial for robust CZSL.
  • Bridging Disentangled and Composed CZSL: The method cleverly combines the strengths of two existing CZSL paradigms: disentangled (separate attribute/object recognition) and composed (joint attribute-object understanding). It uses the latter to inform and improve the former.
  • Dynamic Prototypes: The idea of dynamically adjusting classifier prototypes based on the specific composition is a novel way to enhance discriminative power at the classification stage.

Key Results

The proposed SCD-CZSL method achieves state-of-the-art results on both UT-Zappos and C-GQA datasets.

  • On UT-Zappos, the method achieves an AUC of 31.8±0.1 and an HM of 46.3±0.5, outperforming previous methods like OADis and being comparable to SCEN, while using a simpler, non-generative approach.
  • On C-GQA, the method achieves an AUC of 3.2 and an HM of 14.1, surpassing the previous SOTA (Co-CGE) and reproduced OADis.
  • Ablation studies demonstrate that each proposed module (embedding alignment and prototype modulation) contributes significantly to the overall performance improvement.

Limitations and Future Work

The authors acknowledge a few limitations:

  • Open-Set Application: The method's reliance on pre-defined attribute and object embeddings might limit its performance in truly open-set scenarios where entirely novel concepts could appear. They suggest incorporating word embeddings into the disentangled recognition model training as a potential solution.
  • Consistent Visual Representations: For some concepts, sub-classes might maintain very similar visual representations (e.g., "blue car" vs. "green car" might both look like a "car"). While their method increases the distance between these representations, further exploration into more precise sub-class discrimination is suggested.

Future work could focus on addressing these limitations, particularly by exploring more robust ways to handle open-set scenarios and refining sub-class discrimination techniques.

Summary updated

Abstract

Compositional Zero-Shot Learning (CZSL) aims at identifying unseen compositions composed of previously seen attributes and objects during the test phase. In real images, the visual appearances of attributes and objects (primitive concepts) generally interact with each other. Namely, the visual appearances of an attribute may change when composed with different objects, and vice versa. But previous works overlook this important property. In this paper, we introduce a simple yet effective approach with leveraging sub-class discrimination. Specifically, we define the primitive concepts in different compositions as sub-classes, and then maintain the sub-class discrimination to address the above challenge. More specifically, inspired by the observation that the composed recognition models could account for the differences across sub-classes, we first propose to impose the embedding alignment between the composed and disentangled recognition to incorporate sub-class discrimination at the feature level. Then we develop the prototype modulator networks to adjust the class prototypes w.r.t. the composition information, which can enhance sub-class discrimination at the classifier level. We conduct extensive experiments on the challenging benchmark datasets, and the considerable performance improvement over state-of-the-art approaches is achieved, which indicates the effectiveness of our method. Our code is available at https://github.com/hxm97/SCD-CZSL.
BibTeX
@inproceedings{hu2023leveraging,
  author = {Xiaoming Hu and Zilei Wang},
  title = {{Leveraging Sub-class Discimination for Compositional Zero-Shot Learning}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  url = {https://openreview.net/forum?id=NBEAVhfOSj},
}
Figure