Revisiting Visual Product for Compositional Zero-Shot Learning

Shyamgopal Karthik, Massimiliano Mancini, Zeynep AkataDistShift @ NeurIPS 2021 · OpenReview

TL;DR

This paper revisits the "Visual Product" (VisProd) baseline for Compositional Zero-Shot Learning (CZSL), showing that with increased model capacity and a simple bias correction for seen classes, it can achieve state-of-the-art results. This suggests that simpler, independent object and state prediction might be a more robust approach than complex compositional models, especially in challenging open-world settings.

What Problem Does This Paper Address?

Compositional Zero-Shot Learning (CZSL) is the challenging task of identifying objects and their associated states in images, even when those specific combinations haven't been seen during training. For example, recognizing a "wet dog" when the model has only seen "dry dogs" and "wet cars." The core difficulty lies in generalizing to novel combinations where object appearances can change drastically based on states (e.g., a "pureed tomato" looks very different from a "raw tomato") and states can manifest differently depending on the object. Existing methods often rely on complex architectures and external information like word embeddings to model these relationships. This paper questions whether these complexities are always necessary by re-examining a much simpler baseline.

Methods

The authors revisit the Visual Product (VisProd) method. The core idea of VisProd is to treat CZSL as a multi-task problem: predict the object and the state independently. Given an image, it's first encoded into a feature representation. Then, two separate classifiers (one for objects, one for states) predict the probabilities for each. The joint probability of a composition is then approximated by the product of these independent probabilities.

The paper identifies two key reasons for VisProd's initial poor performance:

  1. Insufficient Network Capacity: Early VisProd implementations used simple linear classifiers, which lacked the power to learn complex object and state representations. The authors address this by using deeper, non-linear classifiers (MLPs).
  2. Bias Towards Seen Classes: CZSL models tend to perform better on combinations seen during training. The authors propose a simple fix: during inference, they subtract a learned scalar bias from the scores of seen compositions. This helps to level the playing field for unseen combinations.

The improved model is referred to as VisProd++ when using deeper MLPs for classifiers.

Datasets

The paper evaluates their approach on three standard CZSL benchmarks:

  • MIT-States: A dataset known for its label noise, particularly in state predictions.
  • UT-Zappos: A dataset of clothing items with attributes.
  • C-GQA: A challenging dataset with a very large number of possible compositions (around 280k).

Key Insights and Ideas

  • Simplicity Can Be Powerful: The paper demonstrates that a simple, independent prediction of objects and states, when implemented with sufficient model capacity and a bias correction, can rival or surpass complex, specialized CZSL architectures.
  • Bias Correction is Crucial: The simple act of penalizing scores for seen compositions significantly improves performance, especially in settings where generalization to unseen combinations is paramount. This highlights the importance of carefully considering evaluation metrics and potential biases in zero-shot learning.
  • Open-World CZSL is a Strong Testbed: VisProd++ shows remarkable performance in the Open-World CZSL setting, where the model has no prior knowledge of the possible unseen compositions. This suggests its independent prediction approach is robust to the vast combinatorial space.

Key Results

  • State-of-the-Art Performance: VisProd++ achieves comparable or superior results to recent state-of-the-art methods on all three tested benchmarks (MIT-States, UT-Zappos, C-GQA) in both closed-world and open-world CZSL settings.
  • Open-World Dominance: On the challenging C-GQA dataset for open-world CZSL, VisProd++ achieves the new state-of-the-art, outperforming previous methods by a significant margin (e.g., achieving an AUC of 0.75 compared to 0.39 and 0.47 for other top methods).
  • Effect of Depth: Ablation studies show that increasing the depth of the object and state classifiers (up to 3 layers) significantly boosts performance, indicating the need for sufficient model capacity.
  • Effect of Bias Correction: The scalar bias correction for seen compositions is shown to be critical, dramatically improving the harmonic mean performance.

Limitations and Future Work

  • Label Noise on MIT-States: The authors note that VisProd++ is somewhat penalized by the inherent label noise in the MIT-States dataset, particularly for state predictions.
  • Independence Assumption: While effective, the core VisProd assumption of independence between object and state prediction might not capture all nuanced interactions.
  • Future Work: The paper hopes to revive interest in visual-only methods for CZSL and encourages further exploration of treating object and state prediction separately as a competitive approach. The success in open-world CZSL suggests potential for further research in handling extremely large output spaces.

Summary updated

Abstract

Compositional Zero-Shot Learning (CZSL) aims to recognize compositions of objects and states in images, and generalize to the unseen compositions of objects and states. Recent works tackled this problem effectively by using side information (e.g., word embeddings) together with either consistency constraints or specific network designs modeling the relationships between objects, states, compositions, and visual features. In this work, we take a step back, and we revisit the simplest baseline for this task, i.e., Visual Product (VisProd). VisProd considers CZSL as a multi-task problem, predicting objects and states separately. Despite its appealing simplicity, this baseline showed low performance in early CZSL studies. Here we identify the two main reasons behind such unimpressive initial results: network capacity and bias on the seen classes. We show that simple modifications to the object and state predictors allow the model to achieve either comparable or superior results w.r.t. the recent state of the art in both the open-world and closed-world CZSL settings on three different benchmarks.
BibTeX
@inproceedings{karthik2021revisiting,
  author = {Shyamgopal Karthik and Massimiliano Mancini and Zeynep Akata},
  title = {{Revisiting Visual Product for Compositional Zero-Shot Learning}},
  booktitle = {DistShift @ NeurIPS},
  year = {2021},
  url = {https://openreview.net/forum?id=Yc9Vh1nn-2I},
}
Figure