Abstract
Traditional Zero-Shot Learning (ZSL) relies on extensive, large-scale attribute annotations for training images, which is often impractical for real-world scalability. This paper introduces a more realistic ZSL setting where only a few images per seen class are annotated, leading to a severe "attribute sparsity" problem. To address this, we propose a unified framework combining Sparse Attribute Propagation (SAP) and Bidirectional Projection Learning (BPL). SAP propagates available annotations to unannotated images using graph-based sparse coding, while BPL utilizes a self-reconstruction regularizer to mitigate the projection domain shift between seen and unseen classes. Our approach achieves state-of-the-art results on standard benchmarks and demonstrates the ability to effectively leverage noisy web images to boost performance.
Introduction
Standard Zero-Shot Learning (ZSL) aims to recognize unseen classes by bridging visual features and semantic descriptors, such as class attributes. However, the heavy reliance on massive annotated datasets for seen classes limits ZSL's applicability in specialized domains like fine-grained or medical image recognition. This paper tackles the challenge of "attribute sparsity," where only a handful of annotated images are available per seen class. In such settings, standard models struggle to learn reliable mappings between visual and semantic spaces.
Proposed Methodology
To overcome attribute sparsity, we propose a unified inductive ZSL framework that integrates two core components: Sparse Attribute Propagation and Bidirectional Projection Learning.
Sparse Attribute Propagation (SAP)
SAP expands the limited pool of annotated data by propagating attributes to a larger set of unannotated images. By employing a graph-based approach with $L_1$-norm regularization (sparse coding), the model suppresses the noise inherent in estimating missing attributes. This allows the framework to utilize unannotated web images, which can be filtered via a pre-trained CNN to serve as a free and scalable data source.
Bidirectional Projection Learning (BPL)
Once attributes are propagated, BPL learns a mapping from visual features to the semantic space and a subsequent mapping back to the visual space. This autoencoder-like structure forces the model to reconstruct the original features, acting as a powerful regularizer that addresses the "projection domain shift"—the discrepancy between the feature distributions of seen and unseen classes.
Experimental Results
We evaluated our framework on four standard ZSL benchmarks: Animals with Attributes (AwA), CUB-200-2011 Birds (CUB), aPascal&Yahoo (aPY), and SUN Attribute (SUN). Additionally, we introduced the CUB+Web dataset to test the model's ability to handle noisy, external web data.
Performance Comparison
Our model consistently outperformed five representative baselines (RPL, ESZSL, SSE, ZSKL, and RN) across all datasets. As shown in the results table, our approach achieved significant improvements, particularly in the most challenging settings with very few annotated images ($K$).
Key Insights
- Web Data Utility: Unannotated web images can effectively substitute for manually curated unannotated data. Sparse coding successfully tames the noise from search engine results, allowing the model to fill semantic gaps.
- Deep Model Limitations: State-of-the-art deep models like Relation Net (RN) were found to struggle in sparse annotation settings, often overfitting to the limited available data. Our propagation technique provides a more robust alternative for these scenarios.
- Generalization to Social Tagging: The SAP formulation is highly adaptable; when applied to Social Image Annotation (SIA) on the NUS-WIDE dataset, it improved F1-scores by over 30% by refining noisy user tags.
Conclusion
By addressing the realistic constraint of sparse attribute annotations, this work provides a scalable path for Zero-Shot Learning. The combination of Sparse Attribute Propagation and Bidirectional Projection Learning not only sets new performance benchmarks but also demonstrates that noisy, unannotated web data can be a valuable asset in training robust semantic-visual mappings.
Summary updated