Abstract
This paper addresses the task of learning an image classifier when some categories are defined by semantic descriptions only (e.g., visual attributes) while others are defined by exemplar images. This task is often referred to as Zero-Shot Classification (ZSC). Most previous methods rely on learning a common embedding space to compare visual features of unknown categories with semantic descriptions. This paper argues that these approaches are limited because efficient discriminative classifiers cannot be used and classification tasks involving both seen and unseen categories (Generalized Zero-Shot Classification or GZSC) cannot be addressed efficiently. In contrast, this paper suggests addressing ZSC and GZSC by learning a conditional generator using seen classes to generate artificial training examples for categories without exemplars. ZSC is then transformed into a standard supervised learning problem. Experiments with four generative models and five datasets validate the approach, yielding state-of-the-art results on both ZSC and GZSC.
1. Introduction
Zero-Shot Classification (ZSC) is the challenge of identifying objects from categories that were never present during training—for example, recognizing a "Zebra" having only seen "Horses" and reading a description of stripes. Traditionally, researchers solved this by learning a common embedding space where visual features and semantic attributes (like "has wings" or "is furry") could be compared directly.
However, this traditional approach has two major flaws. First, it prevents the use of powerful, standard discriminative classifiers like Softmax. Second, it suffers from a significant bias problem in "Generalized" ZSC (GZSC), where the model must distinguish between both seen and unseen classes at test time. Traditional models are heavily biased toward seen classes, often failing to recognize unseen ones entirely.
2. The Generative Approach
The authors propose a paradigm shift: instead of mapping images and text into a shared space, use generative models to "imagine" visual features for classes that have never been seen. By generating synthetic training data for unseen categories, the authors turn the difficult ZSC problem into a standard supervised learning task.
The workflow involves three main steps:
- Training: Train a generator on "seen" classes using their images and attributes.
- Generation: Use the generator to create synthetic visual features for "unseen" classes based on their text descriptions.
- Classification: Train a standard supervised classifier (like a Softmax layer) on a mix of real "seen" data and synthetic "unseen" data.
The authors explore four different generative architectures: Generative Moment Matching Networks (GMMN), Auxiliary Classifier GANs (AC-GAN), Denoising Auto-Encoders (DAE), and Adversarial Auto-Encoders (AAE). Crucially, the model generates high-level visual features (e.g., from VGG or GoogLeNet) rather than raw pixels, which is computationally efficient and focuses on the information needed for classification.
3. Experimental Setup
The approach was evaluated on five benchmarks: AWA (Animals with Attributes), CUB (Caltech-UCSD Birds), SUN Attributes, aP&Y (Apascal & Ayahoo), and a large-scale test on ImageNet involving 1,000 seen and over 20,000 unseen classes.
4. Results and Analysis
The generative approach achieved state-of-the-art performance across all datasets. Notably, the GMMN model emerged as the most stable and consistently high-performing generator, averaging 67.3% accuracy across standard ZSC tasks. The authors attribute this to GMMN's explicit modeling of distribution divergence, which provides more stable training compared to adversarial methods.
As shown in the results for the AWA dataset, the most significant gains were seen in the Generalized ZSC setting. Accuracy for unseen classes in the generalized setting (u → a) jumped from roughly 1-9% in previous methods to 32.32%. By including synthetic examples of unseen classes during training, the model learns the decision boundaries between seen and unseen classes, effectively removing the inherent bias toward seen categories. On the massive ImageNet benchmark, the model improved "Flat-Hit@1" accuracy by 5% over existing competitors in the 2-hop scenario.
5. Conclusion
By reframing Zero-Shot Learning as a data augmentation problem solved through generative modeling, this work overcomes the limitations of embedding-based methods. While the quality of generated features depends on the quality of semantic descriptions, the approach demonstrates remarkable scalability and robustness, particularly in the challenging generalized setting. Future work may explore more sophisticated natural language processing models to capture richer semantic nuances for large-scale tasks.
Summary updated