Zero-shot Learning with Many Classes by High-rank Deep Embedding Networks

Yuchen Guo, Guiguang Ding, Jungong Han, Hang Shao, Xin Lou, Qionghai Dai

Introduction

Zero-shot Learning (ZSL) is designed to recognize objects from classes never seen during training by utilizing auxiliary information, such as text descriptions or semantic attributes. While ZSL has shown promise on small-scale datasets (e.g., 50 classes), its performance significantly degrades when scaled to large-scale benchmarks like ImageNet, which contains over 20,000 classes. This paper identifies a fundamental mathematical bottleneck—the rank limitation—and proposes a novel architecture called GREEN (High-rank Deep Embedding Networks) to overcome it.

The Rank Bottleneck in ZSL

Most existing ZSL models rely on a bilinear compatibility function ($xWy^T$) to match image features ($x$) to class features ($y$). Mathematically, the complexity (rank) of this matching process is constrained by the dimensionality of the feature space. When dealing with tens of thousands of classes, a simple linear mapping lacks the expressive power to capture the fine-grained nuances required for accurate classification. This "underfitting" at scale is the primary reason for the performance drop observed in traditional ZSL methods.

The GREEN Architecture

To bypass the rank limitation, the authors introduce GREEN, which employs a feature-dependent mixture of softmaxes. Instead of a single linear mapping, the model utilizes multiple parallel branches to approximate a high-rank compatibility function.

The GREEN architecture uses multiple branches and a weighting factor to create a high-rank embedding space.

As shown in the architecture diagram above, the process involves:

  1. Multiple Branches: The matching process is split into $B$ different branches, each learning specific compatibility parameters.
  2. Feature-Dependent Gating: A weighting mechanism analyzes the input image to determine which branches are most relevant (e.g., specializing in "bird-like" features vs. "background").
  3. High-Rank Approximation: By summing these weighted non-linear transformations, the model effectively increases the rank of the compatibility function, allowing it to capture both macro-level categories and micro-level details.

The researchers developed two variants: GREEN-S (Shallow), using pre-extracted features, and GREEN-D (Deep), which fine-tunes the entire network end-to-end.

Experimental Results and Scaling

The model was evaluated on the SUN dataset and various subsets of ImageNet (ranging from popular classes to the long-tail "least popular" classes). GREEN consistently outperformed state-of-the-art (SOTA) methods. For instance, on the full ImageNet dataset, GREEN-D achieved 2.06% accuracy—more than double the performance of baselines like SYNC (0.96%) and DEVISE (0.49%).

A critical finding of the study is the relationship between the number of branches ($B$) and model performance. As the number of branches increases, the model's ability to distinguish between many classes improves, confirming the high-rank hypothesis.

Accuracy across various datasets improves as the number of branches (B) increases, demonstrating the benefit of high-rank embeddings.

Conclusion

GREEN addresses the scalability issues of Zero-shot Learning by breaking the rank barrier inherent in bilinear models. By using a mixture of softmaxes, the model achieves a more expressive embedding space capable of handling the complexity of thousands of unseen classes. While performance on long-tail classes remains a challenge, GREEN represents a significant step forward in making ZSL viable for real-world, large-scale applications.

Summary updated

Abstract

Zero-shot learning (ZSL) is a recently emerging research topic which aims to build classification models for unseen classes with knowledge from auxiliary seen classes. Though many ZSL works have shown promising results on small-scale datasets by utilizing a bilinear compatibility function, the ZSL performance on large-scale datasets with many classes (say, ImageNet) is still unsatisfactory. We argue that the bilinear compatibility function is a low-rank approximation of the true compatibility function such that it is not expressive enough especially when there are a large number of classes because of the rank limitation. To address this issue, we propose a novel approach, termed as High-rank Deep Embedding Networks (GREEN), for ZSL with many classes. In particular, we propose a feature-dependent mixture of softmaxes as the image-class compatibility function, which is a simple extension of the bilinear compatibility function, but yields much better results. It utilizes a mixture of non-linear transformations with feature-dependent latent variables to approximate the true function in a high-rank way, which makes GREEN more expressive. Experiments on several datasets including ImageNet demonstrate GREEN significantly outperforms the state-of-the-art approaches.
BibTeX
@inproceedings{guo2019zero,
  author = {Yuchen Guo and Guiguang Ding and Jungong Han and Hang Shao and Xin Lou and Qionghai Dai},
  title = {{Zero-shot Learning with Many Classes by High-rank Deep Embedding Networks}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2019},
  doi = {10.24963/ijcai.2019/337},
}
Figure