Abstract
Generalized Zero-Shot Learning (GZSL) models often fail when encountering "unseen" test classes that bear little resemblance to the "seen" training classes. This performance drop stems from a reliance on the similarity assumption, where models expect semantic and visual overlap between training and testing domains. We introduce DSECN (Diverse Semantics from External Class Names), a framework that utilizes external class names and hierarchical taxonomies to build a more robust semantic-visual mapping. DSECN demonstrates state-of-the-art performance in a new "Practical Setting," achieving a +35.14% improvement on dissimilar classes in the CUB dataset while training in less than 40 seconds.
1. The Challenge of Dissimilar Classes in GZSL
Zero-Shot Learning (ZSL) aims to recognize objects from classes not seen during training by leveraging semantic attributes. Generalized ZSL (GZSL) extends this by requiring the model to distinguish between both seen and unseen classes at inference. However, current GZSL models suffer from a "dissimilarity gap": they perform well when unseen classes are similar to seen ones (e.g., different bird species) but fail catastrophically when faced with entirely different categories, such as recognizing a building after training only on birds.
To address this, we propose a more realistic evaluation framework that includes both similar and dissimilar unseen classes. This "Practical Setting" highlights the limitations of existing state-of-the-art (SOTA) methods, which often drop below 5% accuracy in dissimilar contexts because the mapping learned from seen classes does not generalize to distant semantic spaces.
2. The DSECN Framework
The proposed DSECN framework enhances the robustness of the semantic-to-visual mapping through three integrated components:
- Visual Flow: A core sub-network that maps class attributes to visual features.
- Diverse Semantic Enhancement (DSE): To prevent overfitting to training classes, DSECN incorporates thousands of external class names from large-scale datasets like ImageNet-21K. These names are anchored in visual space using a frozen, pre-trained classification head, encouraging a more universal mapping.
- Hierarchy Taxonomy Enhancement (HTE): By leveraging WordNet, the model exploits hierarchical "is-a" relationships (hypernyms and hyponyms). This provides a structured semantic scaffold, helping the model generalize from specific seen classes (e.g., "Chihuahua") to broader categories (e.g., "Dog").
3. Experimental Results and Efficiency
DSECN was evaluated on standard benchmarks (CUB, AWA2, SUN) using a "Practical Setting" that combines similar and dissimilar unseen classes.
Performance Gains: As shown in the results below, DSECN provides a transformative boost for dissimilar classes. On the CUB dataset, the harmonic mean accuracy rose from 12.30% to 47.44%, a staggering +35.14% improvement over previous methods like DGZ.
Efficiency: Unlike complex generative models (GANs or VAEs) that require hours of training, DSECN is highly efficient. For example, on the AWA2 dataset, DSECN trains in just 37.9 seconds, compared to 9.5 hours for generative-based competitors like ZLA. This makes it a highly practical "plug-and-play" solution for existing GZSL pipelines.
4. Conclusion
DSECN demonstrates that exploring diverse external semantics is a powerful and efficient alternative to complex feature synthesis for GZSL. By regularizing the semantic-visual mapping with a wide array of class names and hierarchical structures, the model achieves unprecedented robustness against dissimilar unseen classes. While the current approach is most effective for global embedding methods, future work will focus on extending these benefits to local, part-based recognition tasks.
Summary updated