Boosting Zero-shot Learning via Contrastive Optimization of Attribute Representations

Yu Du, Miaojing Shi, Fangyun Wei, Guoqi LiIEEE TNNLS 2022 · OpenReview

TL;DR

This paper introduces CoAR-ZSL, a novel framework for zero-shot learning (ZSL) that tackles the challenge of attribute representation by explicitly learning "attribute prototypes" beyond individual images. By contrastively optimizing these prototypes with image-specific attribute features, CoAR-ZSL significantly boosts recognition accuracy for unseen classes.

What Problem Does This Paper Address?

Zero-shot learning (ZSL) is a fascinating area of machine learning that aims to enable models to recognize objects from classes they haven't seen during training. The core idea is to leverage semantic information, often in the form of attributes (like "has wings," "is furry"), to bridge the gap between seen and unseen classes. Many ZSL methods learn an embedding function that maps visual features to semantic representations. A recent trend is to extract rich "attribute features" directly from images. However, a key limitation of these approaches is that attribute features extracted from individual images can be scattered in the embedding space due to intra-attribute variance. This makes them less representative of the attribute itself, hindering generalization. CoAR-ZSL addresses this by explicitly learning more robust attribute representations.

Methods

The proposed CoAR-ZSL framework introduces two main innovations for attribute representation:

  • Attribute Prototypes: Instead of relying solely on image-extracted features, CoAR-ZSL learns "attribute prototypes." These are learned representations of attributes that exist independently of any specific image. A dedicated "prototype generation module" takes attribute semantics (e.g., one-hot encoded attribute vectors) and generates these prototypes.
  • Contrastive Optimization: The framework then employs a contrastive learning strategy to optimize both the image-specific attribute-level features and the learned attribute prototypes. This involves:
  • Attribute-level Triplet Loss (Lattp): This loss pulls image-specific attribute features closer to their corresponding attribute prototypes while pushing them away from other attribute prototypes.
  • Hard Example-Based Contrastive Loss (Lattf): To further enhance robustness, this loss pulls attribute-level features from different images that share the same attribute closer together, while pushing apart features from different attributes. Crucially, it focuses on "hard examples" – those that are close but not identical – to better handle intra-attribute variance.

The framework also learns class-level features and optimizes them against class prototypes, similar to prior work. CoAR-ZSL is flexible and can be built upon either a CNN-based (ResNet101) or a Transformer-based (ViT) backbone.

Datasets

The paper evaluates CoAR-ZSL on three widely-used ZSL benchmarks:

  • Caltech-UCSD Birds-200-2011 (CUB): A fine-grained dataset with 312 attributes.
  • SUN Attribute Database (SUN): A large scene dataset with 102 attributes.
  • Animals with Attributes 2 (AwA2): A coarser dataset with 85 attributes.

The authors follow standard train/test splits for these datasets.

Key Insights and Ideas

  • Attribute Prototypes are Key: The core insight is that learning explicit, image-agnostic attribute prototypes significantly improves the robustness and representativeness of attribute features, leading to better ZSL performance. This moves beyond just learning from individual image instances.
  • Contrastive Learning for Attributes: Applying contrastive learning not just at the class level but specifically for attribute-level features and prototypes is a powerful way to enforce desired relationships in the embedding space.
  • Hard Examples Matter: Focusing contrastive loss on "hard examples" (those that are challenging to distinguish) is more effective than using all examples, as it pushes the model to learn finer distinctions.
  • Transformer Advantage: While CNNs are effective, the Vision Transformer (ViT) backbone offers superior performance, highlighting its potential for ZSL tasks.

Key Results

CoAR-ZSL achieves state-of-the-art results on all three benchmark datasets for both ZSL and Generalized ZSL (GZSL) settings.

  • CNN Backbone: Achieves top-1 accuracy (T1) of 79.2% on CUB, 66.7% on SUN, and 74.1% on AwA2. For GZSL, it achieves generalized harmonic mean (ACCH) of 74.0% on CUB, 43.4% on SUN, and 73.2% on AwA2.
  • Transformer Backbone (CoAR-ZSL*): Further improves performance, reaching T1 of 79.9% on CUB, 79.4% on SUN, and 78.7% on AwA2. ACCH scores are 74.4% on CUB, 61.0% on SUN, and 82.0% on AwA2. This significantly outperforms previous transformer-based ZSL methods like ViT-ZSL.
  • Qualitative Results: Visualizations show that CoAR-ZSL accurately localizes attributes within images and that its learned attribute features are well-clustered in the embedding space, unlike those from prior methods.

Limitations and Future Work

  • Reliance on Human-Crafted Attributes: A significant limitation is the dependence on pre-defined, human-annotated attributes. This restricts its applicability to datasets lacking such information. The authors suggest exploring methods to learn shared prototypes across classes to simulate attributes.
  • Computational Cost of Transformers: While ViT-based CoAR-ZSL* achieves better accuracy, it comes with increased memory usage, inference time, and pre-training costs compared to the CNN-based version.
  • Future Directions: The authors suggest exploring graph neural networks to encode attribute relations for more sophisticated optimization and potentially learning shared prototypes to overcome the reliance on explicit attributes.

Summary updated

Abstract

Zero-shot learning (ZSL) aims to recognize classes that do not have samples in the training set. One representative solution is to directly learn an embedding function associating visual features with corresponding class semantics for recognizing new classes. Many methods extend upon this solution, and recent ones are especially keen on extracting rich features from images, e.g. attribute features. These attribute features are normally extracted within each individual image; however, the common traits for features across images yet belonging to the same attribute are not emphasized. In this paper, we propose a new framework to boost ZSL by explicitly learning attribute prototypes beyond images and contrastively optimizing them with attribute-level features within images. Besides the novel architecture, two elements are highlighted for attribute representations: a new prototype generation module is designed to generate attribute prototypes from attribute semantics; a hard example-based contrastive optimization scheme is introduced to reinforce attribute-level features in the embedding space. We explore two alternative backbones, CNN-based and transformer-based, to build our framework and conduct experiments on three standard benchmarks, CUB, SUN, AwA2. Results on these benchmarks demonstrate that our method improves the state of the art by a considerable margin. Our codes will be available at https://github.com/dyabel/CoAR-ZSL.git
BibTeX
@article{du2022boosting,
  author = {Yu Du and Miaojing Shi and Fangyun Wei and Guoqi Li},
  title = {{Boosting Zero-shot Learning via Contrastive Optimization of Attribute Representations}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2022},
  url = {https://openreview.net/forum?id=mkzhT5pHxj},
}
Figure