Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning

Zhi Chen, Zi Huang, Jingjing Li, Zheng Zhang

TL;DR

This paper introduces a novel framework for Generalized Zero-Shot Learning (GZSL) that tackles the "distribution overlap" problem—where models struggle to distinguish between seen and unseen classes in a shared feature space. By combining dual Variational Autoencoders (VAEs) with a triplet loss for sharper feature discrimination and an entropy-based calibration step, the authors significantly reduce classification uncertainty and achieve state-of-the-art results across multiple benchmarks.

What Problem Does This Paper Address?

In Zero-Shot Learning (ZSL), a model must recognize objects it has never seen during training (e.g., identifying a "Zebra" having only seen "Horses" and "Stripes"). Generalized ZSL (GZSL) is the more realistic, harder version where the test set contains both seen and unseen classes.

The authors identify two major hurdles in current GZSL:

  1. Distribution Overlap: Generative models often overfit seen classes, causing the synthesized features for unseen classes to "overlap" with seen ones. This creates a "zone of uncertainty" where the model can't tell if a sample is a poorly recognized seen object or a new unseen one.
  2. Sparse Training Samples: Many datasets have very few examples per class, making it difficult for generative models (like GANs or VAEs) to learn the complex, high-dimensional manifold of visual features.

Methods

The proposed framework consists of two main stages: Generative Metric Learning (GML) and Entropy-based Uncertainty Calibration (EUC).

1. Generative Metric Learning (GML)

The authors build upon the CADA-VAE architecture, using two VAEs (one for visual features, one for semantic attributes) that map to a shared latent space.

  • The Innovation: They incorporate a triplet loss into the latent space. Standard VAEs might cluster different classes too closely. The triplet loss explicitly forces the model to minimize the distance between samples of the same class while maximizing the distance between different classes. This creates a more "discriminative" latent space where class boundaries are clearer.

2. Entropy-based Uncertainty Calibration (EUC)

To handle the overlap issue at test time, the authors propose a "cascade" classification strategy:

  • Entropy as a Confidence Metric: When a classifier is unsure, the "entropy" of its prediction is high (the probability is spread thin across many classes).
  • The Filter: They first pass test samples through a general classifier. If the entropy is low, the model is confident it's a "seen" class and uses a specialized seen-class classifier. If the entropy is high, the sample is treated as "uncertain" (likely unseen) and is passed to the broader GZSL classifier.

Datasets

The model was evaluated on six standard ZSL benchmarks:

  • Coarse-grained: aPaY (Attribute Pascal and Yahoo), AWA1, and AWA2 (Animals with Attributes).
  • Fine-grained: CUB (Birds), SUN (Scenes), and FLO (Oxford Flowers).
  • Video Datasets: The authors also extended their testing to video classification using HMDB51 and UCF101.

Key Insights and Ideas

  • Discriminative Latent Spaces Matter: Simply aligning visual and semantic spaces isn't enough. By adding metric learning (triplet loss), the model learns why classes are different, not just how they relate to attributes.
  • Entropy is a Powerful Heuristic: The paper demonstrates that unseen classes naturally produce higher entropy in a model trained primarily on seen distributions. Using a simple threshold to "gate" these samples prevents the common GZSL failure mode of misclassifying unseen objects as seen ones.
  • Soft Margins vs. Fixed Pivots: Unlike previous methods that tried to force features toward a fixed "class mean" (visual pivot), the triplet loss allows for a more natural, flexible distribution that still maintains clear inter-class margins.

Key Results

The method achieved significant performance gains, particularly in the Harmonic Mean (H), which balances accuracy for both seen and unseen classes:

  • aPaY Dataset: Improved the Harmonic Mean from 40.3% (previous SOTA) to 44.9%.
  • Seen Class Accuracy: On the aPaY dataset, the seen class accuracy jumped from 51.8% to 62.7% specifically due to the entropy calibration.
  • ZSL Performance: In conventional ZSL (unseen only), the model set new records on CUB (61.73%) and FLO (67.2%).
  • Ablation Studies: Removing the triplet loss dropped accuracy by about 2%, proving that the metric learning component is vital for feature quality.

Limitations and Future Work

  • Threshold Sensitivity: The entropy threshold used to distinguish seen from unseen samples is currently a hyper-parameter that must be tuned manually for each dataset. The authors note that performance is "very sensitive" to this value.
  • Minor Trade-off: The uncertainty calibrator slightly decreases accuracy for unseen classes (by a fraction of a percent) because a small portion of novel samples are "trapped" and misclassified as seen. However, the massive boost in seen-class accuracy far outweighs this "tiny sacrifice."
  • Future Direction: The authors suggest developing an automated mechanism for tuning the entropy threshold dynamically.

Summary updated

Abstract

Compared to conventional zero-shot learning (ZSL) where recognising unseen classes is the primary or only aim, the goal of generalized zero-shot learning (GZSL) is to recognise both seen and unseen classes. Most GZSL methods typically learn to synthesise visual representations from semantic information on the unseen classes. However, these types of models are prone to overfitting the seen classes, resulting in distribution overlap between the generated features of the seen and unseen classes. The overlapping region is filled with uncertainty as the model struggles to determine whether a test case from within the overlap is seen or unseen. Further, these generative methods suffer in scenarios with sparse training samples. The models struggle to learn the distribution of high dimensional visual features and, therefore, fail to capture the most discriminative inter-class features. To address these issues, in this paper, we propose a novel framework that leverages dual variational autoencoders with a triplet loss to learn discriminative latent features and applies the entropy-based calibration to minimize the uncertainty in the overlapped area between the seen and unseen classes. Specifically, the dual generative model with the triplet loss synthesises inter-class discriminative latent features that can be mapped from either visual or semantic space. To calibrate the uncertainty for seen classes, we calculate the entropy over the softmax probability distribution from a general classifier. With this approach, recognising the seen samples within the seen classes is relatively straightforward, and there is less risk that a seen sample will be misclassified into an unseen class in the overlapped region. Extensive experiments on six benchmark datasets demonstrate that the proposed method outperforms state-of-the-art approaches.
BibTeX
@inproceedings{chen2021entropy,
  author = {Zhi Chen and Zi Huang and Jingjing Li and Zheng Zhang},
  title = {{Entropy-Based Uncertainty Calibration for Generalized Zero-Shot Learning}},
  booktitle = {ADC},
  year = {2021},
  doi = {10.1007/978-3-030-69377-0\_12},
}

Loading related content...

Figure