Abstract
Zero-Shot Learning (ZSL) aims to recognize novel object categories by leveraging semantic descriptions to bridge the gap between seen and unseen classes. While generative models have recently improved ZSL by synthesizing visual features for unseen classes, these generated features often lack the stability and discriminative power of real data. This paper introduces LisGAN (Leveraging Invariant Side GAN), a novel framework that anchors the generative process using "soul samples"—prototypical representations of class centroids. By ensuring synthesized features remain tethered to these stable invariants and employing a multi-stage cascade classifier, LisGAN achieves state-of-the-art performance across multiple benchmarks.
Introduction
The primary challenge in Zero-Shot Learning is the lack of training data for target classes. Traditional approaches attempted to map visual features directly to a semantic space (such as attributes or word embeddings), but this often leads to the "hubness problem" and significant information loss. Recent shifts toward Generative ZSL use Generative Adversarial Networks (GANs) to transform semantic attributes into synthetic visual features, effectively turning ZSL into a standard supervised learning problem.
However, GANs in ZSL are often unconstrained. Because a single text description can correspond to a wide variety of visual appearances, the generator may produce "soulless" features that are noisy or unrepresentative. LisGAN addresses this by focusing on the "invariant side" of the data—the core, stable characteristics that define a category regardless of noise or viewpoint.
Methodology: The LisGAN Framework
LisGAN builds upon the conditional Wasserstein GAN (WGAN) architecture, introducing two primary innovations designed to improve the quality and utility of synthesized features.
1. Soul Samples and Invariant Regularization
To prevent the generator from producing unrealistic features, the authors introduce the concept of soul samples. A soul sample is defined as the meta-representation or centroid of a class's features.
- Multi-View Representation: For seen classes, the model employs $k$-means clustering to identify multiple soul samples per category. This accounts for the "multi-view" nature of objects, recognizing that a single category (e.g., a "horse") may have distinct visual clusters based on pose or orientation.
- Regularization Loss: The model implements two regularization losses ($L_{R1}$ and $L_{R2}$). These force every generated feature to be spatially close to at least one real soul sample of its corresponding class. This ensures that while the generator accounts for variance (the "varying side"), the output remains anchored to the core characteristics (the "invariant side") of the class.
2. The Cascade Classifier
In the inference phase, LisGAN utilizes a two-stage Cascade Classifier to handle the inherent difficulty of Generalized Zero-Shot Learning (GZSL):
- Stage 1 (Confidence Estimation): A standard softmax classifier predicts the class of a test sample and calculates the entropy of the prediction. Low entropy indicates high confidence.
- Stage 2 (Coarse-to-Fine Refinement): Samples identified as high-confidence are used as "anchors." The model then classifies more ambiguous, high-entropy samples by leveraging these anchors to refine the decision boundaries in a coarse-to-fine manner.
Experimental Evaluation
Datasets and Setup
LisGAN was evaluated on five gold-standard computer vision benchmarks: aPaY (Attribute Pascal and Yahoo), AwA (Animals with Attributes), CUB (Caltech-UCSD Birds), FLO (Oxford Flowers), and SUN (Scene UNderstanding). To ensure a fair comparison with existing literature, all experiments utilized ResNet-101 features as the visual backbone.
Results and Discussion
LisGAN consistently outperformed previous state-of-the-art (SOTA) generative methods, such as f-CLSWGAN and GAZSL.
- Standard ZSL: The model achieved 43.1% accuracy on aPaY (a 2% improvement over previous SOTA) and 70.6% on AwA.
- Generalized ZSL (GZSL): In the GZSL setting—where the test set contains both seen and unseen classes—LisGAN demonstrated an average improvement of 2.2% in Harmonic Mean across all datasets.
- Ablation Studies: Results indicated that the "soul sample" regularization was the most critical component for success, significantly stabilizing GAN training and ensuring that synthesized features were representative of their semantic descriptions.
Conclusion
LisGAN demonstrates that the key to effective generative ZSL lies in balancing the diversity of generated samples with the stability of class invariants. By anchoring the GAN with soul samples and utilizing a cascade classification strategy, the model effectively mitigates the domain shift and noise issues prevalent in previous generative approaches. Future work may explore semantic data augmentation to further refine the descriptions of unseen categories, potentially reducing the model's dependency on the quality of initial attribute sets.
Summary updated