Abstract
Compositional Zero-Shot Learning (CZSL) aims to recognize novel combinations of known primitives, such as attributes and objects. A primary challenge is the distribution shift between training and testing, where models often over-confidently misclassify unseen compositions as seen ones. This paper proposes the COMO (Composing Outliers) framework. By synthesizing artificial "outlier" feature representations during training, COMO teaches the model to explicitly distinguish between seen and unseen compositions. This distinction enables the application of specialized recognition strategies at test time, leading to state-of-the-art performance on standard CZSL benchmarks.
The Challenge of Compositional Zero-Shot Learning
The core objective of CZSL is to recognize new combinations of known "primitives." For instance, a model trained on "young cat" and "wet dog" should ideally be able to recognize a "wet cat." However, the visual manifestation of an attribute (e.g., "wet") can vary significantly depending on the object it modifies (e.g., dog vs. cat), a phenomenon known as distribution shift.
In generalized CZSL settings, where the test set includes both seen and unseen pairs, models frequently struggle with overconfidence, mislabeling novel compositions as familiar ones. The authors of COMO argue that if a model can first determine whether an image contains a seen or unseen composition, it can then employ a more effective, specialized recognition strategy.
The COMO Framework
The COMO architecture utilizes a frozen CLIP backbone and incorporates two primary classification modules: Composition Classification, which assesses the holistic compatibility of an image with an (attribute, object) pair, and Primitive Classification, which independently evaluates the image against the attribute and object.
Key Methodological Innovations
- Outlier Composing: To simulate unseen compositions during training, the authors synthesize "fake" feature vectors. By adding the attribute representation from one training image to the object representation of another, they create proxies for out-of-distribution (OOD) data.
- Outlier Exposure: The model is explicitly trained to treat these synthesized features as OOD, forcing a low-confidence (uniform) output for such outliers.
- Inference Strategy: At test time, the confidence of the composition classifier serves as a "seen/unseen" detector. Based on this detection, the model applies different weights ($\lambda_s$ for seen, $\lambda_u$ for unseen) to fuse the outputs of the composition and primitive classifiers.
- Loss Constraints: A disentanglement loss prevents attribute and object features from leaking into one another, while a discriminability loss ensures that synthesized features remain semantically grounded.
Experimental Results
COMO was evaluated on three major CZSL benchmarks: MIT-States (diverse attributes/objects), UT-Zappos (fine-grained shoe types), and C-GQA (complex visual scenes). The model achieved state-of-the-art (SOTA) results, particularly in the Area Under the Curve (AUC) metric, which measures the balance between seen and unseen class accuracy.
| Dataset | Metric | COMO Performance |
|---|---|---|
| MIT-States | AUC | 22.4 |
| UT-Zappos | AUC | 43.7 |
| C-GQA | AUC | SOTA |
Notably, COMO demonstrated significant robustness in open-world settings, where the search space for possible labels is much larger. This suggests that outlier-based training effectively prepares the model for the vast variety of possible unseen combinations.
Discussion and Future Work
The success of COMO highlights that synthesized OOD data can effectively serve as a proxy for real unseen images in feature space. Furthermore, the results confirm that seen and unseen compositions benefit from different recognition logic: seen pairs rely more on holistic patterns, while unseen pairs are better recognized by analyzing primitives independently.
Future improvements could involve automating the selection of hyperparameters like the seen/unseen threshold and fusion weights. Moving from static weights to a neural module that predicts sample-specific scalars could provide even more granular control and further boost performance.
Summary updated