Information Bottleneck Constrained Latent Bidirectional Embedding for Zero-Shot Learning

Yang Liu, Lei Zhou, Xiao Bai, Lin Gu, Tatsuya Harada, Jun ZhouarXiv 2020 · OpenReview

TL;DR

This paper introduces a novel approach to Zero-Shot Learning (ZSL) by using an "information bottleneck" to create a unified latent space that better aligns visual and semantic information. This method significantly improves the ability to recognize unseen object classes, even outperforming state-of-the-art generative models.

What Problem Does This Paper Address?

Zero-Shot Learning (ZSL) aims to enable machines to recognize objects from classes they haven't seen during training, much like humans can. This is typically achieved by transferring knowledge from "seen" classes (with labeled examples) to "unseen" classes (without examples) using semantic descriptions (like attributes). However, existing methods often struggle with:

  • Calibration Deviation and Hubness: Directly mapping high-dimensional visual features to semantic spaces can lead to misalignments and a phenomenon called "hubness," where a few visual features become nearest neighbors to many semantic concepts, hindering generalization.
  • Seen-Unseen Bias: Generative ZSL methods, which create synthetic features for unseen classes, often still suffer from bias because they are trained primarily on seen data.
  • Non-Semantic Information: Visual features contain a lot of information that isn't directly relevant to semantic understanding, making it hard to align them accurately with semantic descriptions.

This paper tackles these challenges by proposing a more robust way to bridge the gap between visual and semantic representations.

Methods

The core of the proposed method is a bidirectional embedding generative model built upon a modified Variational Autoencoder (VAE). Here's a breakdown of the key novelties:

  • Unified Latent Space: The model learns a single latent space where both visual and semantic features are embedded. This aims to create a shared representation that facilitates better alignment.
  • Information Bottleneck (IB) Constraint: This is a key innovation. The authors introduce an IB loss to the latent space. The goal is to maximize the mutual information between the latent space and the semantic attributes while minimizing the mutual information between the latent space and the raw visual features. This effectively forces the model to retain only the essential semantic information from the visual input, discarding irrelevant details.
  • Data Uncertainty Estimation & Bias Passing: To handle noise in visual data and the inherent variability of visual samples for a single semantic concept, the model incorporates uncertainty estimation to learn data biases. A "bias passing" mechanism then shares this bias with the latent semantic distribution to improve alignment.
  • Wake-Sleep Procedure: To leverage both real and generated data for training, a wake-sleep procedure is employed. In the "wake" phase, real visual data is used to train feature representation. In the "sleep" phase, generated visual data is used to train feature abstraction, leading to more disentangled latent features.
  • Transductive ZSL Extension: The framework is extended to Transductive ZSL (TZSL) by using the generated latent features to assign pseudo-labels to unseen images. A robust loss is then introduced to handle the inherent label noise in this process.

Datasets

The proposed method was evaluated on four widely used ZSL benchmark datasets:

  • CUB-200-2011 (CUB)
  • SUN attribute (SUN)
  • Animals with Attributes 2 (AWA2)
  • Attributes Pascal and Yahoo (aPY)

The paper uses the "Proposed Split (PS)" for dividing classes into seen and unseen categories, which is a standard practice for fair comparison.

Key Insights and Ideas

  • Information Bottleneck for ZSL is Powerful: Applying the information bottleneck principle to ZSL is a novel and effective way to disentangle semantic information from visual features, leading to better generalization.
  • Tight Visual-Semantic Coupling is Crucial: Simply generating features isn't enough; the alignment between visual and semantic representations needs to be tightly controlled, which the unified latent space and IB constraint achieve.
  • Handling Data Noise is Key: Real-world data is noisy. The paper effectively addresses this through uncertainty estimation and a robust loss for the TZSL extension.
  • Wake-Sleep Improves Abstraction: The wake-sleep procedure is vital for improving the model's ability to abstract and generate disentangled latent representations.

Key Results

The paper reports significant improvements across various ZSL settings:

  • Generalized ZSL (GZSL): The proposed method consistently outperforms state-of-the-art methods on the harmonic mean (H) metric across CUB, AWA2, and aPY datasets. For example, on CUB, the proposed method achieves an H of 54.1, compared to the next best at 52.3.
  • Conventional ZSL: The method also achieves the best results on three out of four datasets for conventional ZSL.
  • Transductive ZSL (TZSL): The extended TZSL method demonstrates competitive performance, achieving the highest GZSL and conventional ZSL results on CUB and strong second-place finishes on AWA2 and SUN.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores the influence of hyperparameters like q (for GCE loss) and r (for pseudo-labeling in TZSL), optimal tuning might still be dataset-dependent. The authors found q=0.4 and r=1.4 to be good trade-offs for CUB.
  • Computational Cost: Generative models and VAEs can be computationally intensive. While not explicitly detailed, the wake-sleep procedure and bidirectional embedding likely add to this.
  • Future Work: The paper suggests that the framework can be further extended to other ZSL variants and potentially applied to more complex tasks requiring fine-grained visual-semantic understanding. The effectiveness of the IB constraint suggests exploring its application in other areas of representation learning.

Summary updated

Abstract

Zero-shot learning (ZSL) aims to recognize novel classes by transferring semantic knowledge from seen classes to unseen classes. Though many ZSL methods rely on a direct mapping between the visual and the semantic space, the calibration deviation and hubness problem limit the generalization capability to unseen classes. Recently emerged generative ZSL methods generate unseen image features to transform ZSL into a supervised classification problem. However, most generative models still suffer from the seen-unseen bias problem as only seen data is used for training. To address these issues, we propose a novel bidirectional embedding based generative model with a tight visual-semantic coupling constraint. We learn a unified latent space that calibrates the embedded parametric distributions of both visual and semantic spaces. Since the embedding from high-dimensional visual features comprise much non-semantic information, the alignment of visual and semantic in latent space would inevitably been deviated. Therefore, we introduce information bottleneck (IB) constraint to ZSL for the first time to preserve essential attribute information during the mapping. Specifically, we utilize the uncertainty estimation and the wake-sleep procedure to alleviate the feature noises and improve model abstraction capability. In addition, our method can be easily extended to transductive ZSL setting by generating labels for unseen images. We then introduce a robust loss to solve this label noise problem. Extensive experimental results show that our method outperforms the state-of-the-art methods in different ZSL settings on most benchmark datasets. The code will be available at https://github.com/osierboy/IBZSL.
BibTeX
@misc{liu2020information,
  author = {Yang Liu and Lei Zhou and Xiao Bai and Lin Gu and Tatsuya Harada and Jun Zhou},
  title = {{Information Bottleneck Constrained Latent Bidirectional Embedding for Zero-Shot Learning}},
  year = {2020},
  eprint = {2009.07451},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.07451},
}
Figure