Zero-shot Learning with Complementary Attributes

Xiaofeng Xu, Ivor W. Tsang, Chuancai LiuarXiv 2018 · arXiv · OpenReview

TL;DR

This paper introduces "complementary attributes" (CA) to improve zero-shot learning (ZSL) by providing a more comprehensive semantic representation of objects. For ZSL methods that rely on attribute independence, they propose a novel "rank aggregation" framework to overcome this unrealistic assumption. Together, these contributions significantly boost ZSL performance across various benchmarks.

What Problem Does This Paper Address?

Traditional object recognition systems excel when trained on large datasets of labeled images. However, they struggle when encountering objects from categories not seen during training. Zero-shot learning (ZSL) aims to bridge this gap by enabling models to recognize novel object classes using high-level semantic information, typically in the form of attributes. Attributes describe properties of objects (e.g., "has wings," "is furry").

The core challenge ZSL faces is effectively transferring knowledge from seen classes to unseen classes. While attributes are crucial for this transfer, existing ZSL methods often treat all attributes equally, even if some are irrelevant to specific objects. This can lead to suboptimal semantic representations. Furthermore, some ZSL approaches, particularly those based on probability prediction, make a strong and often unrealistic assumption that attributes are independent of each other. This paper tackles these two limitations: improving the richness of attribute representations and addressing the attribute independence assumption.

Methods

The paper introduces two main innovations:

  • Complementary Attributes (CA): The authors propose augmenting the original set of attributes with "complementary attributes." These are essentially the logical negation of the original attributes (e.g., if "has tail" is an original attribute, "has no tail" is its complementary attribute). The intuition is that while an original attribute might be irrelevant for a particular object (e.g., "small" for an elephant), its complement ("not small") might be highly relevant. By including both, the model gains a more complete and discriminative semantic description of objects, enhancing the ability to transfer knowledge to unseen classes. CA can be easily integrated by simply expanding the attribute representation space.
  • Rank Aggregation (RA) Framework: For ZSL methods that rely on predicting attribute probabilities (like the Probability-Prediction Strategy, PPZSL), a key assumption is that attributes are independent. This is often not true in reality. To overcome this, the authors propose a novel rank aggregation framework. Instead of directly combining attribute probabilities, this framework first ranks the potential unseen classes based on each attribute's prediction. Then, it aggregates these ranks to produce a final ranking of classes, effectively circumventing the independence assumption and leading to more robust predictions.

Datasets

The proposed methods were evaluated on several standard ZSL benchmark datasets:

  • Animal with Attribute (AwA)
  • Animal with Attribute 2 (AwA2)
  • attribute-Pascal-Yahoo (aPY)
  • Caltech-UCSD Bird 200-2011 (CUB)
  • SUN Attribute Database (SUN)

Additionally, experiments were conducted on the large-scale ImageNet dataset, using various splits to test performance on a much broader range of classes.

Key Insights and Ideas

  • Attributes are not created equal: The paper highlights that the relevance of attributes varies significantly across object classes. Simply using all available attributes without considering their specific relevance can dilute the semantic signal.
  • Negation is powerful: Introducing the logical negation of attributes (complementary attributes) provides a simple yet effective way to capture more nuanced semantic information, especially for attributes that might be less informative in their original form.
  • Attribute independence is a bottleneck: The assumption of attribute independence in some ZSL models is a significant limitation that hinders performance. The proposed rank aggregation framework offers a principled way to bypass this assumption.
  • Synergy of CA and RA: The combination of complementary attributes and rank aggregation proves to be more powerful than either method alone, demonstrating a synergistic effect in improving ZSL performance.

Key Results

  • Significant Accuracy Gains: The proposed complementary attributes (CA) consistently improved the performance of existing ZSL methods across five benchmark datasets. For example, the mean class accuracy increased by an average of 2.95% across various methods.
  • State-of-the-Art Performance: The combined approach of complementary attributes and rank aggregation achieved state-of-the-art results on several datasets for both conventional and generalized ZSL tasks.
  • Robustness: The confusion matrices showed that the proposed methods not only improved accuracy but also reduced prediction uncertainties, indicating a more robust model. For instance, on the AwA dataset, the proposed method significantly improved performance on classes that were previously difficult to classify.
  • ImageNet Performance: The methods also showed substantial improvements on the large-scale ImageNet dataset, demonstrating their scalability.
  • Theoretical Guarantees: Theoretical analysis suggested that complementary attributes can improve the PAC-style generalization bound of ZSL models, providing a principled justification for their use.

Limitations and Future Work

  • Attribute Independence Assumption: While the rank aggregation framework addresses the attribute independence assumption for PPZSL, the paper acknowledges that attributes themselves are designed to describe objects, and their inherent correlations are a fundamental aspect of semantic representation.
  • Future Work: The authors suggest exploring the fusion of multi-source auxiliary information (beyond attributes) to further enhance the discriminative ability of semantic representations. They also mention that their work is concentrated on attributes, and other forms of assistant information could be similarly expanded with complementary formulations.

Summary updated

Abstract

Zero-shot learning (ZSL) aims to recognize unseen objects using disjoint seen objects via sharing attributes. The generalization performance of ZSL is governed by the attributes, which transfer semantic information from seen classes to unseen classes. To take full advantage of the knowledge transferred by attributes, in this paper, we introduce the notion of complementary attributes (CA), as a supplement to the original attributes, to enhance the semantic representation ability. Theoretical analyses demonstrate that complementary attributes can improve the PAC-style generalization bound of original ZSL model. Since the proposed CA focuses on enhancing the semantic representation, CA can be easily applied to any existing attribute-based ZSL methods, including the label-embedding strategy based ZSL (LEZSL) and the probability-prediction strategy based ZSL (PPZSL). In PPZSL, there is a strong assumption that all the attributes are independent of each other, which is arguably unrealistic in practice. To solve this problem, a novel rank aggregation framework is proposed to circumvent the assumption. Extensive experiments on five ZSL benchmark datasets and the large-scale ImageNet dataset demonstrate that the proposed complementary attributes and rank aggregation can significantly and robustly improve existing ZSL methods and achieve the state-of-the-art performance.
BibTeX
@misc{xu2018zero,
  author = {Xiaofeng Xu and Ivor W. Tsang and Chuancai Liu},
  title = {{Zero-shot Learning with Complementary Attributes}},
  year = {2018},
  eprint = {1804.06505},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1804.06505},
}
Figure