Abstract
Compared to conventional zero-shot learning (ZSL) where recognising unseen classes is the primary or only aim, the goal of generalized zero-shot learning (GZSL) is to recognise both seen and unseen classes. Most GZSL methods typically learn to synthesise visual representations from semantic information on the unseen classes. However, these types of models are prone to overfit the seen classes, resulting in distribution overlap between the generated features of the seen and unseen classes. The overlapping region is filled with uncertainty as the model struggles to determine whether a test case from within the overlap is seen or unseen. Further, these generative methods suffer in scenarios with sparse training samples. The models struggle to learn the distribution of high dimensional visual features and, therefore, fail to capture the most discriminative inter-class features. To address these issues, in this paper, we propose a novel framework that leverages dual variational autoencoders with a triplet loss to learn discriminative latent features and applies the entropy-based calibration to minimize the uncertainty in the overlapped area between the seen and unseen classes. Specifically, the dual generative model with the triplet loss synthesises inter-class discriminative latent features that can be mapped from either visual or semantic space. To calibrate the uncertainty for seen classes, we calculate the entropy over the softmax probability distribution from a general classifier. Extensive experiments on six benchmark datasets demonstrate that the proposed method outperforms state-of-the-art approaches.
Introduction
Generalized Zero-Shot Learning (GZSL) presents a significant challenge in computer vision: a model must recognize objects from classes it has never encountered during training while simultaneously maintaining performance on seen classes. This is inherently more difficult than standard ZSL, where the test set only contains unseen classes.
Current GZSL approaches face two primary hurdles. First, distribution overlap occurs when generative models overfit seen classes, causing synthesized features for unseen classes to overlap with seen ones in the feature space. This creates a "zone of uncertainty" where the model struggles to distinguish between a poorly recognized seen object and a novel unseen one. Second, sparse training samples in many datasets make it difficult for generative models (like GANs or VAEs) to learn the complex, high-dimensional manifold of visual features, often failing to capture discriminative inter-class characteristics.
Proposed Methodology
The proposed framework addresses these issues through two main components: Generative Metric Learning (GML) and Entropy-based Uncertainty Calibration (EUC).
Generative Metric Learning (GML)
Building upon the CADA-VAE architecture, the framework utilizes two Variational Autoencoders (VAEs)—one for visual features and one for semantic attributes—that map inputs into a shared latent space.
The key innovation here is the incorporation of a triplet loss within the latent space. While standard VAEs might cluster different classes too closely, the triplet loss explicitly forces the model to minimize the distance between samples of the same class (anchor and positive) while maximizing the distance between different classes (anchor and negative). This creates a more discriminative latent space with clearer class boundaries, helping the model learn why classes are different rather than just how they relate to attributes.
Entropy-based Uncertainty Calibration (EUC)
To mitigate the distribution overlap at test time, the authors introduce a "cascade" classification strategy based on entropy.
Entropy serves as a powerful heuristic for confidence. When a classifier is uncertain, the entropy of its softmax probability distribution is high. The EUC process works as follows:
- Test samples are first passed through a general classifier to generate probability vectors.
- If the entropy is low (below a threshold), the model is confident the sample belongs to a "seen" class and routes it to a specialized seen-class classifier.
- If the entropy is high (above a threshold), the sample is treated as "uncertain" (likely unseen) and is passed to the broader GZSL classifier that considers both seen and unseen classes.
This gating mechanism prevents the common GZSL failure mode of misclassifying unseen objects as seen ones.
Experimental Results
The model was evaluated on six standard ZSL benchmarks, including coarse-grained datasets (aPaY, AWA1, AWA2) and fine-grained datasets (CUB, SUN, FLO), as well as video datasets (HMDB51, UCF101).
The method achieved significant performance gains, particularly in the Harmonic Mean (H), which balances accuracy for both seen and unseen classes:
- aPaY Dataset: Improved the Harmonic Mean from 40.3% to 44.9%.
- Seen Class Accuracy: On aPaY, seen class accuracy increased from 51.8% to 62.7% due to entropy calibration.
- ZSL Performance: In conventional ZSL, the model set new records on CUB (61.73%) and FLO (67.2%).
- Ablation Studies: Removing the triplet loss resulted in an accuracy drop of approximately 2%, confirming the importance of metric learning for feature quality.
Conclusion and Future Work
This research demonstrates that combining discriminative latent space learning with uncertainty calibration significantly improves GZSL performance. By using triplet loss to create clearer inter-class margins and entropy to gate uncertain samples, the model effectively navigates the distribution overlap problem.
However, the entropy threshold is currently a manually tuned hyper-parameter, and performance is sensitive to its value. Future work will focus on developing automated mechanisms to tune this threshold dynamically, further robustifying the framework for diverse real-world applications.
Summary updated