Learning semantic ambiguities for zero-shot learning

Celina Hanouti, Hervé Le Borgne

TL;DR

This paper introduces a novel regularization technique for zero-shot learning (ZSL) that combats the bias towards seen classes in generative ZSL models. By training generative models to synthesize features for "ambiguous" semantic prototypes (created by mixing existing class descriptions), the method improves the model's ability to distinguish between unseen classes, achieving state-of-the-art results on several benchmarks.

What Problem Does This Paper Address?

Zero-shot learning (ZSL) aims to classify objects from classes that were not seen during training. This is typically achieved by learning a mapping between visual features of "seen" classes and their corresponding semantic descriptions (like attributes or word embeddings). A common approach involves using generative models to synthesize visual features for unseen classes based on their semantic prototypes. However, these generative models often inherit a bias from the seen classes, as they are primarily trained on visual examples of those classes. This bias can lead to poor performance, especially in the more realistic "generalized ZSL" (GZSL) setting, where the model must classify both seen and unseen classes. The paper tackles this problem by proposing a method to regularize generative ZSL models to reduce this bias and improve discrimination between unseen classes.

Methods

The core contribution is a new regularization loss that can be applied to existing conditional generative ZSL models. Instead of relying solely on real visual data from seen classes, the proposed method focuses on the semantic space. It constructs "virtual ambiguous classes" by linearly interpolating the semantic prototypes of existing classes. The generative model is then trained to synthesize visual features for these ambiguous prototypes. This process forces the generator to learn more discriminative features that can better differentiate between potentially similar unseen classes, effectively learning the "transitions" between categories. This regularization is independent of specific visual instances and can be integrated into various generative ZSL frameworks, such as f-VAEGAN-D2 and TF-VAEGAN.

Datasets

The paper evaluates its method on four standard ZSL datasets:

  • Caltech UCSD Birds 200-2011 (CUB): Fine-grained classification of bird species.
  • SUN Attribute dataset (SUN): Scene recognition with attribute labels.
  • Oxford Flowers (FLO): Fine-grained classification of flower species.
  • Animals with Attributes 2 (AWA2): Coarse-grained classification of animal species with attributes.

The evaluation uses the "proposed splits" protocol, which ensures that the test classes are not present in the ImageNet dataset used for pre-training visual features.

Key Insights and Ideas

  • Semantic Interpolation is Key: Mixing semantic descriptions (attributes, word embeddings) makes more sense than mixing raw images, as visual inconsistencies can be problematic. This allows the model to learn meaningful transitions between concepts.
  • Regularizing Ambiguity: By training the generator to synthesize features for "ambiguous" or "frontier" prototypes, the model learns to better distinguish between distinct classes, reducing the bias inherited from seen classes.
  • Generality of the Approach: The proposed regularization loss is designed to be model-agnostic, meaning it can be easily integrated into various conditional generative ZSL architectures, enhancing their performance.
  • Improved Discrimination: The method specifically targets improving the discriminative power of synthesized features, which is crucial for distinguishing between fine-grained or semantically similar unseen classes.

Key Results

The proposed method achieves state-of-the-art results on several benchmarks for both inductive and transductive ZSL and GZSL settings.

  • On the CUB dataset, the method achieved a significant improvement in inductive GZSL, reaching 67.2% accuracy, up from 56.9%.
  • In transductive ZSL, the approach established a new state-of-the-art on CUB, SUN, and AWA2 datasets with an mNRG score of 22.8.
  • The regularization consistently improved performance across datasets, with improvements ranging from 1.5 to 3 points when integrated into TF-VAEGAN.
  • The improvements in GZSL are primarily driven by better recognition of unseen classes, while performance on seen classes is either maintained or slightly decreased, indicating a successful reduction in bias.

Limitations and Future Work

  • Linear Interpolation: The current method uses linear interpolation to create ambiguous prototypes. The authors suggest exploring non-linear interpolation or constructing ambiguous classes from more than two real classes as future work.
  • Extrapolation: Experiments with extrapolation (using negative lambda values) did not yield significant improvements, suggesting that interpolation is more effective.
  • Bias Still Exists: While the method reduces the bias towards seen classes, it doesn't entirely eliminate it, as seen class scores often remain higher than unseen class scores in GZSL.
  • Broader Applicability: The authors note that this approach could be beneficial for other tasks involving ambiguous visual and semantic information, such as multimodal entity linking and retrieval.

Summary updated

Abstract

Zero-shot learning (ZSL) aims at recognizing classes for which no visual sample is available at training time. To address this issue, one can rely on a semantic description of each class. A typical ZSL model learns a mapping between the visual samples of seen classes and the corresponding semantic descriptions, in order to do the same on unseen classes at test time. State of the art approaches rely on generative models that synthesize visual features from the prototype of a class, such that a classifier can then be learned in a supervised manner. However, these approaches are usually biased towards seen classes whose visual instances are the only one that can be matched to a given class prototype. We propose a regularization method that can be applied to any conditional generative-based ZSL method, by leveraging only the semantic class prototypes. It learns to synthesize discriminative features for possible semantic description that are not available at training time, that is the unseen ones. The approach is evaluated for ZSL and GZSL on four datasets commonly used in the literature, either in inductive or transductive settings, with results on-par or above state of the art approaches. The code is available at https://github.com/hanouticelina/lsa-zsl.
BibTeX
@article{hanouti2023learning,
  author = {Celina Hanouti and Hervé Le Borgne},
  title = {{Learning semantic ambiguities for zero-shot learning}},
  journal = {Multimedia Tools and Applications},
  year = {2023},
  url = {https://openreview.net/forum?id=X4En0AA7DR},
}
Figure