Abstract
Contemporary state-of-the-art approaches to Zero-Shot Learning (ZSL) train generative nets to synthesize examples conditioned on the provided metadata. Thereafter, classifiers are trained on these synthetic data in a supervised manner. In this work, we introduce Z2FSL, an end-to-end generative ZSL framework that uses such an approach as a backbone and feeds its synthesized output to a Few-Shot Learning (FSL) algorithm. The two modules are trained jointly. Z2FSL solves the ZSL problem with a FSL algorithm, reducing, in effect, ZSL to FSL. A wide class of algorithms can be integrated within our framework. Our experimental results show consistent improvement over several baselines. The proposed method, evaluated across standard benchmarks, shows state-of-the-art or competitive performance in ZSL and Generalized ZSL tasks.
Introduction
In deep learning, the challenge of Zero-Shot Learning (ZSL) arises when a model encounters categories unseen during training. Traditional state-of-the-art ZSL methods typically employ a two-stage process: first, training a generative model (such as a GAN or VAE) to "hallucinate" synthetic feature vectors for unseen classes based on auxiliary attributes; second, training a static classifier on this synthetic data.
However, this decoupled approach has limitations. The generator often lacks direct feedback on whether its synthetic samples are discriminative enough for classification. Furthermore, static classifiers are rigid and require retraining when class sets change. This paper introduces Z2FSL, a framework that conceptualizes ZSL as a Few-Shot Learning (FSL) problem where the "few examples" are provided by a generator.
Methods: The Z2FSL Framework
The core innovation of Z2FSL is the end-to-end coupling of a generative backbone with a dynamic FSL classifier, specifically Prototypical Networks.
The framework consists of:
- The Backbone: A generative model (e.g., f-VAEGAN) that synthesizes image features from class attributes and noise.
- The FSL Classifier: A Prototypical Network that assigns query samples to classes by comparing them to a "support set" of generated prototypes.
- End-to-End Training: The FSL classifier’s loss is backpropagated to the generator. This forces the generator to produce samples that are not just statistically accurate but specifically discriminative, aiding class separability.
The total objective function combines the generative loss ($\mathcal{L}_{ZSL}$) and the FSL loss ($\mathcal{L}_{FSL}$): $$\mathcal{L}_{Z2FSL} = \mathcal{L}_{ZSL} + \gamma\mathcal{L}_{FSL}$$
Experimental Setup
The authors evaluated Z2FSL on three standard ZSL benchmarks using ResNet-101 features:
- CUB (Caltech UCSD Birds 200): Fine-grained bird classification (312 attributes).
- AwA2 (Animals with Attributes 2): 50 animal categories (85 attributes).
- SUN (Scene Classification): 717 scene categories (102 attributes).
Results and Discussion
Z2FSL consistently outperformed its generative backbones across all datasets. By treating the generator's output as a "support set," the model leverages the inherent flexibility of FSL algorithms to handle new classes dynamically.
Key findings include:
- Performance Gains: On the SUN dataset, Z2FSL achieved a state-of-the-art top-1 accuracy of 66.5%.
- Backbone Agnosticism: Integrating Z2FSL with VAE or WGAN backbones resulted in absolute accuracy improvements ranging from approximately 1.5% to over 8% depending on the dataset and backbone.
- Generalized ZSL (GZSL): The method showed strong performance in the GZSL setting, notably achieving a harmonic mean score of 66.8 on AwA2.
Conclusion
Z2FSL effectively reduces Zero-Shot Learning to an end-to-end Few-Shot Learning task. While the model shows significant promise, challenges remain regarding seen-class bias in generalized settings and optimization stability during fine-tuning. Future work may explore more robust pre-training strategies to further mitigate these biases.
Summary updated