Zero-Shot Learning with Sparse Attribute Propagation

Nanyi Fei, Jiajun Liu, Jiechao Guan, Zhiwu Lu, Tao Xiang, Ji-Rong WenarXiv 2018 · arXiv

Abstract

Traditional Zero-Shot Learning (ZSL) relies on extensive, large-scale attribute annotations for training images, which is often impractical for real-world scalability. This paper introduces a more realistic ZSL setting where only a few images per seen class are annotated, leading to a severe "attribute sparsity" problem. To address this, we propose a unified framework combining Sparse Attribute Propagation (SAP) and Bidirectional Projection Learning (BPL). SAP propagates available annotations to unannotated images using graph-based sparse coding, while BPL utilizes a self-reconstruction regularizer to mitigate the projection domain shift between seen and unseen classes. Our approach achieves state-of-the-art results on standard benchmarks and demonstrates the ability to effectively leverage noisy web images to boost performance.

Introduction

Standard Zero-Shot Learning (ZSL) aims to recognize unseen classes by bridging visual features and semantic descriptors, such as class attributes. However, the heavy reliance on massive annotated datasets for seen classes limits ZSL's applicability in specialized domains like fine-grained or medical image recognition. This paper tackles the challenge of "attribute sparsity," where only a handful of annotated images are available per seen class. In such settings, standard models struggle to learn reliable mappings between visual and semantic spaces.

Proposed Methodology

To overcome attribute sparsity, we propose a unified inductive ZSL framework that integrates two core components: Sparse Attribute Propagation and Bidirectional Projection Learning.

Sparse Attribute Propagation (SAP)

SAP expands the limited pool of annotated data by propagating attributes to a larger set of unannotated images. By employing a graph-based approach with $L_1$-norm regularization (sparse coding), the model suppresses the noise inherent in estimating missing attributes. This allows the framework to utilize unannotated web images, which can be filtered via a pre-trained CNN to serve as a free and scalable data source.

Bidirectional Projection Learning (BPL)

Once attributes are propagated, BPL learns a mapping from visual features to the semantic space and a subsequent mapping back to the visual space. This autoencoder-like structure forces the model to reconstruct the original features, acting as a powerful regularizer that addresses the "projection domain shift"—the discrepancy between the feature distributions of seen and unseen classes.

Figure 1: Overview of the proposed framework, illustrating the propagation of attributes from a few annotated images to unannotated web images via SAP, followed by BPL.

Experimental Results

We evaluated our framework on four standard ZSL benchmarks: Animals with Attributes (AwA), CUB-200-2011 Birds (CUB), aPascal&Yahoo (aPY), and SUN Attribute (SUN). Additionally, we introduced the CUB+Web dataset to test the model's ability to handle noisy, external web data.

Performance Comparison

Our model consistently outperformed five representative baselines (RPL, ESZSL, SSE, ZSKL, and RN) across all datasets. As shown in the results table, our approach achieved significant improvements, particularly in the most challenging settings with very few annotated images ($K$).

Figure 6: Comparison of ZSL accuracy (%) against state-of-the-art baselines across four datasets with varying numbers of annotated images per class (K).

Key Insights

  • Web Data Utility: Unannotated web images can effectively substitute for manually curated unannotated data. Sparse coding successfully tames the noise from search engine results, allowing the model to fill semantic gaps.
  • Deep Model Limitations: State-of-the-art deep models like Relation Net (RN) were found to struggle in sparse annotation settings, often overfitting to the limited available data. Our propagation technique provides a more robust alternative for these scenarios.
  • Generalization to Social Tagging: The SAP formulation is highly adaptable; when applied to Social Image Annotation (SIA) on the NUS-WIDE dataset, it improved F1-scores by over 30% by refining noisy user tags.

Conclusion

By addressing the realistic constraint of sparse attribute annotations, this work provides a scalable path for Zero-Shot Learning. The combination of Sparse Attribute Propagation and Bidirectional Projection Learning not only sets new performance benchmarks but also demonstrates that noisy, unannotated web data can be a valuable asset in training robust semantic-visual mappings.

Summary updated

Abstract

Zero-shot learning (ZSL) aims to recognize a set of unseen classes without any training images. The standard approach to ZSL requires a set of training images annotated with seen class labels and a semantic descriptor for seen/unseen classes (attribute vector is the most widely used). Class label/attribute annotation is expensive; it thus severely limits the scalability of ZSL. In this paper, we define a new ZSL setting where only a few annotated images are collected from each seen class. This is clearly more challenging yet more realistic than the conventional ZSL setting. To overcome the resultant image-level attribute sparsity, we propose a novel inductive ZSL model termed sparse attribute propagation (SAP) by propagating attribute annotations to more unannotated images using sparse coding. This is followed by learning bidirectional projections between features and attributes for ZSL. An efficient solver is provided, together with rigorous theoretic algorithm analysis. With our SAP, we show that a ZSL training dataset can now be augmented by the abundant web images returned by image search engine, to further improve the model performance. Moreover, the general applicability of SAP is demonstrated on solving the social image annotation (SIA) problem. Extensive experiments show that our model achieves superior performance on both ZSL and SIA.
BibTeX
@misc{fei2018zero,
  author = {Nanyi Fei and Jiajun Liu and Jiechao Guan and Zhiwu Lu and Tao Xiang and Ji-Rong Wen},
  title = {{Zero-Shot Learning with Sparse Attribute Propagation}},
  year = {2018},
  eprint = {1812.04427},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1812.04427},
}
Figure