TL;DR
This paper introduces a novel regularization technique for zero-shot learning (ZSL) that combats the bias towards seen classes in generative ZSL models. By training generative models to synthesize features for "ambiguous" semantic prototypes (created by mixing existing class descriptions), the method improves the model's ability to distinguish between unseen classes, achieving state-of-the-art results on several benchmarks.
What Problem Does This Paper Address?
Zero-shot learning (ZSL) aims to classify objects from classes that were not seen during training. This is typically achieved by learning a mapping between visual features of "seen" classes and their corresponding semantic descriptions (like attributes or word embeddings). A common approach involves using generative models to synthesize visual features for unseen classes based on their semantic prototypes. However, these generative models often inherit a bias from the seen classes, as they are primarily trained on visual examples of those classes. This bias can lead to poor performance, especially in the more realistic "generalized ZSL" (GZSL) setting, where the model must classify both seen and unseen classes. The paper tackles this problem by proposing a method to regularize generative ZSL models to reduce this bias and improve discrimination between unseen classes.
Methods
The core contribution is a new regularization loss that can be applied to existing conditional generative ZSL models. Instead of relying solely on real visual data from seen classes, the proposed method focuses on the semantic space. It constructs "virtual ambiguous classes" by linearly interpolating the semantic prototypes of existing classes. The generative model is then trained to synthesize visual features for these ambiguous prototypes. This process forces the generator to learn more discriminative features that can better differentiate between potentially similar unseen classes, effectively learning the "transitions" between categories. This regularization is independent of specific visual instances and can be integrated into various generative ZSL frameworks, such as f-VAEGAN-D2 and TF-VAEGAN.
Datasets
The paper evaluates its method on four standard ZSL datasets:
- Caltech UCSD Birds 200-2011 (CUB): Fine-grained classification of bird species.
- SUN Attribute dataset (SUN): Scene recognition with attribute labels.
- Oxford Flowers (FLO): Fine-grained classification of flower species.
- Animals with Attributes 2 (AWA2): Coarse-grained classification of animal species with attributes.
The evaluation uses the "proposed splits" protocol, which ensures that the test classes are not present in the ImageNet dataset used for pre-training visual features.
Key Insights and Ideas
- Semantic Interpolation is Key: Mixing semantic descriptions (attributes, word embeddings) makes more sense than mixing raw images, as visual inconsistencies can be problematic. This allows the model to learn meaningful transitions between concepts.
- Regularizing Ambiguity: By training the generator to synthesize features for "ambiguous" or "frontier" prototypes, the model learns to better distinguish between distinct classes, reducing the bias inherited from seen classes.
- Generality of the Approach: The proposed regularization loss is designed to be model-agnostic, meaning it can be easily integrated into various conditional generative ZSL architectures, enhancing their performance.
- Improved Discrimination: The method specifically targets improving the discriminative power of synthesized features, which is crucial for distinguishing between fine-grained or semantically similar unseen classes.
Key Results
The proposed method achieves state-of-the-art results on several benchmarks for both inductive and transductive ZSL and GZSL settings.
- On the CUB dataset, the method achieved a significant improvement in inductive GZSL, reaching 67.2% accuracy, up from 56.9%.
- In transductive ZSL, the approach established a new state-of-the-art on CUB, SUN, and AWA2 datasets with an mNRG score of 22.8.
- The regularization consistently improved performance across datasets, with improvements ranging from 1.5 to 3 points when integrated into TF-VAEGAN.
- The improvements in GZSL are primarily driven by better recognition of unseen classes, while performance on seen classes is either maintained or slightly decreased, indicating a successful reduction in bias.
Limitations and Future Work
- Linear Interpolation: The current method uses linear interpolation to create ambiguous prototypes. The authors suggest exploring non-linear interpolation or constructing ambiguous classes from more than two real classes as future work.
- Extrapolation: Experiments with extrapolation (using negative lambda values) did not yield significant improvements, suggesting that interpolation is more effective.
- Bias Still Exists: While the method reduces the bias towards seen classes, it doesn't entirely eliminate it, as seen class scores often remain higher than unseen class scores in GZSL.
- Broader Applicability: The authors note that this approach could be beneficial for other tasks involving ambiguous visual and semantic information, such as multimodal entity linking and retrieval.
Summary updated