Improving Generalized Zero-Shot Learning by Exploring the Diverse Semantics from External Class Names

Yapeng Li, Yong Luo, Zengmao Wang, Bo DuCVPR 2024 · OpenReview

Abstract

Generalized Zero-Shot Learning (GZSL) models often fail when encountering "unseen" test classes that bear little resemblance to the "seen" training classes. This performance drop stems from a reliance on the similarity assumption, where models expect semantic and visual overlap between training and testing domains. We introduce DSECN (Diverse Semantics from External Class Names), a framework that utilizes external class names and hierarchical taxonomies to build a more robust semantic-visual mapping. DSECN demonstrates state-of-the-art performance in a new "Practical Setting," achieving a +35.14% improvement on dissimilar classes in the CUB dataset while training in less than 40 seconds.

1. The Challenge of Dissimilar Classes in GZSL

Zero-Shot Learning (ZSL) aims to recognize objects from classes not seen during training by leveraging semantic attributes. Generalized ZSL (GZSL) extends this by requiring the model to distinguish between both seen and unseen classes at inference. However, current GZSL models suffer from a "dissimilarity gap": they perform well when unseen classes are similar to seen ones (e.g., different bird species) but fail catastrophically when faced with entirely different categories, such as recognizing a building after training only on birds.

To address this, we propose a more realistic evaluation framework that includes both similar and dissimilar unseen classes. This "Practical Setting" highlights the limitations of existing state-of-the-art (SOTA) methods, which often drop below 5% accuracy in dissimilar contexts because the mapping learned from seen classes does not generalize to distant semantic spaces.

2. The DSECN Framework

The proposed DSECN framework enhances the robustness of the semantic-to-visual mapping through three integrated components:

  1. Visual Flow: A core sub-network that maps class attributes to visual features.
  2. Diverse Semantic Enhancement (DSE): To prevent overfitting to training classes, DSECN incorporates thousands of external class names from large-scale datasets like ImageNet-21K. These names are anchored in visual space using a frozen, pre-trained classification head, encouraging a more universal mapping.
  3. Hierarchy Taxonomy Enhancement (HTE): By leveraging WordNet, the model exploits hierarchical "is-a" relationships (hypernyms and hyponyms). This provides a structured semantic scaffold, helping the model generalize from specific seen classes (e.g., "Chihuahua") to broader categories (e.g., "Dog").
DSECN architecture overview showing training and inference phases.

3. Experimental Results and Efficiency

DSECN was evaluated on standard benchmarks (CUB, AWA2, SUN) using a "Practical Setting" that combines similar and dissimilar unseen classes.

Performance Gains: As shown in the results below, DSECN provides a transformative boost for dissimilar classes. On the CUB dataset, the harmonic mean accuracy rose from 12.30% to 47.44%, a staggering +35.14% improvement over previous methods like DGZ.

Comparison of evaluation settings and performance gains on dissimilar classes.

Efficiency: Unlike complex generative models (GANs or VAEs) that require hours of training, DSECN is highly efficient. For example, on the AWA2 dataset, DSECN trains in just 37.9 seconds, compared to 9.5 hours for generative-based competitors like ZLA. This makes it a highly practical "plug-and-play" solution for existing GZSL pipelines.

4. Conclusion

DSECN demonstrates that exploring diverse external semantics is a powerful and efficient alternative to complex feature synthesis for GZSL. By regularizing the semantic-visual mapping with a wide array of class names and hierarchical structures, the model achieves unprecedented robustness against dissimilar unseen classes. While the current approach is most effective for global embedding methods, future work will focus on extending these benefits to local, part-based recognition tasks.

Summary updated

Abstract

Generalized Zero-Shot Learning (GZSL) methods often assume that the unseen classes are similar to seen classes, and thus perform poor when unseen classes are dissimilar to seen classes. Although some existing GZSL approaches can alleviate this issue by leveraging additional semantic information from test unseen classes, their generalization ability to dissimilar unseen classes is still unsatisfactory. This motivates us to study GZSL in the more practical setting, where unseen classes can be either similar or dissimilar to seen classes. In this paper, we propose a simple yet effective GZSL framework by exploring diverse semantics from external class names (DSECN), which is simultaneously robust on the similar and dissimilar unseen classes. This is achieved by introducing diverse semantics from external class names and aligning the introduced semantics to visual space using the classification head of pretrained network. Furthermore, we show that the design idea of DSECN can easily be integrate into other advanced GZSL approaches, such as the generative-based ones, and enhance their robustness for dissimilar unseen classes. Extensive experiments in the practical setting including both similar and dissimilar unseen classes show that our method significantly outperforms the state-of-the-art approaches on all datasets and can be trained very efficiently.
BibTeX
@inproceedings{li2024improving,
  author = {Yapeng Li and Yong Luo and Zengmao Wang and Bo Du},
  title = {{Improving Generalized Zero-Shot Learning by Exploring the Diverse Semantics from External Class Names}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  url = {https://openreview.net/forum?id=Y3vmKkXswo},
}
Figure