Seeing the Unseen: Composing Outliers for Compositional Zero-Shot Learning

Chenchen Jing, Mingyu Liu, Hao Chen, Yuling Xi, Xingyuan Bu, Dong Gong, Chunhua Shen

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize novel combinations of known primitives, such as attributes and objects. A primary challenge is the distribution shift between training and testing, where models often over-confidently misclassify unseen compositions as seen ones. This paper proposes the COMO (Composing Outliers) framework. By synthesizing artificial "outlier" feature representations during training, COMO teaches the model to explicitly distinguish between seen and unseen compositions. This distinction enables the application of specialized recognition strategies at test time, leading to state-of-the-art performance on standard CZSL benchmarks.

The Challenge of Compositional Zero-Shot Learning

The core objective of CZSL is to recognize new combinations of known "primitives." For instance, a model trained on "young cat" and "wet dog" should ideally be able to recognize a "wet cat." However, the visual manifestation of an attribute (e.g., "wet") can vary significantly depending on the object it modifies (e.g., dog vs. cat), a phenomenon known as distribution shift.

Overview of seen vs. unseen compositions and the proposed recognition pipeline.

In generalized CZSL settings, where the test set includes both seen and unseen pairs, models frequently struggle with overconfidence, mislabeling novel compositions as familiar ones. The authors of COMO argue that if a model can first determine whether an image contains a seen or unseen composition, it can then employ a more effective, specialized recognition strategy.

The COMO Framework

The COMO architecture utilizes a frozen CLIP backbone and incorporates two primary classification modules: Composition Classification, which assesses the holistic compatibility of an image with an (attribute, object) pair, and Primitive Classification, which independently evaluates the image against the attribute and object.

The COMO training and inference architecture, highlighting outlier composing and classification modules.

Key Methodological Innovations

  • Outlier Composing: To simulate unseen compositions during training, the authors synthesize "fake" feature vectors. By adding the attribute representation from one training image to the object representation of another, they create proxies for out-of-distribution (OOD) data.
  • Outlier Exposure: The model is explicitly trained to treat these synthesized features as OOD, forcing a low-confidence (uniform) output for such outliers.
  • Inference Strategy: At test time, the confidence of the composition classifier serves as a "seen/unseen" detector. Based on this detection, the model applies different weights ($\lambda_s$ for seen, $\lambda_u$ for unseen) to fuse the outputs of the composition and primitive classifiers.
  • Loss Constraints: A disentanglement loss prevents attribute and object features from leaking into one another, while a discriminability loss ensures that synthesized features remain semantically grounded.

Experimental Results

COMO was evaluated on three major CZSL benchmarks: MIT-States (diverse attributes/objects), UT-Zappos (fine-grained shoe types), and C-GQA (complex visual scenes). The model achieved state-of-the-art (SOTA) results, particularly in the Area Under the Curve (AUC) metric, which measures the balance between seen and unseen class accuracy.

Dataset Metric COMO Performance
MIT-States AUC 22.4
UT-Zappos AUC 43.7
C-GQA AUC SOTA

Notably, COMO demonstrated significant robustness in open-world settings, where the search space for possible labels is much larger. This suggests that outlier-based training effectively prepares the model for the vast variety of possible unseen combinations.

Discussion and Future Work

The success of COMO highlights that synthesized OOD data can effectively serve as a proxy for real unseen images in feature space. Furthermore, the results confirm that seen and unseen compositions benefit from different recognition logic: seen pairs rely more on holistic patterns, while unseen pairs are better recognized by analyzing primitives independently.

Future improvements could involve automating the selection of hyperparameters like the seen/unseen threshold and fusion weights. Moving from static weights to a neural module that predicts sample-specific scalars could provide even more granular control and further boost performance.

Summary updated

Abstract

Compositional zero-shot learning (CZSL) is to recognize unseen attribute-object compositions by learning from seen compositions. The distribution shift between unseen compositions and seen compositions poses challenges to CZSL models, especially when test images are mixed with both seen and unseen compositions. The challenge will be addressed more easily if a model can distinguish unseen/seen compositions and treat them with specific recognition strategies. However, identifying images with unseen compositions is non-trivial, considering that unseen compositions are absent in training and usually contain only subtle differences from seen compositions. In this paper, we propose a novel compositional zero-shot learning method called COMO, which composes outliers in training for distinguishing seen and unseen compositions and further applying specific strategies for them. Specifically, we compose attribute-object representations for unseen compositions based on primitive representations of training images as outliers to enable the model to identify unseen compositions in inference. At test time, the method distinguishes images containing seen/unseen compositions and uses different weights for composition classification and primitive classification to recognize seen/unseen compositions. Experimental results on three datasets show the effectiveness of our method in both the closed-world setting and the open-world setting.
BibTeX
@inproceedings{jing2025seeing,
  author = {Chenchen Jing and Mingyu Liu and Hao Chen and Yuling Xi and Xingyuan Bu and Dong Gong and Chunhua Shen},
  title = {{Seeing the Unseen: Composing Outliers for Compositional Zero-Shot Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/143},
}
Figure