Two-Level Adversarial Visual-Semantic Coupling for Generalized Zero-shot Learning

Shivam Chandhok, Vineeth N. BalasubramanianWACV 2021 · OpenReview

TL;DR

This paper introduces TACO, a novel two-level adversarial framework for Generalized Zero-Shot Learning (GZSL) that enhances visual-semantic coupling. By jointly training generative and inference networks with a novel adversarial joint maximization loss, TACO generates more discriminative features and improves recognition performance on unseen classes.

What Problem Does This Paper Address?

Generalized Zero-Shot Learning (GZSL) aims to classify images from both seen and unseen classes, a challenging task because models are typically trained on limited labeled data. Humans excel at this by transferring knowledge from high-level descriptions to visual recognition. Existing GZSL methods often rely on generative models to synthesize features for unseen classes or use representation learning. However, these approaches can suffer from a lack of tight coupling between visual and semantic information, leading to suboptimal performance. This paper addresses the need for a more robust method that can effectively transfer knowledge across domains and capture the complex distributions of data, especially in the more realistic GZSL setting where test images can belong to either seen or unseen classes.

Methods

The proposed TACO framework unifies generative and inference-based approaches within a two-level adversarial learning strategy.

  • Two-Level Adversarial Training: TACO employs adversarial training for both the generative module (to synthesize image features from semantic attributes) and the inference module (to infer semantic attributes from image features).
  • Joint Maximization Loss: A novel adversarial joint maximization loss is introduced to explicitly encourage a strong coupling between the visual and semantic spaces. This is achieved through a "joint discriminator" that learns to distinguish between real visual-semantic pairs and generated ones.
  • Wasserstein Semantic Alignment Loss: To ensure that generated semantic features are distributionally aligned with real semantic features, a Wasserstein alignment loss is used. This helps to preserve semantic information and reduce loss in the semantic space.
  • Unified Recognition Module: Unlike previous methods that solely use synthesized features, TACO's recognition module leverages both the generated features from the generator and the discriminative latent representations from the inference network. This provides richer information for the final classifier.

Datasets

The authors evaluate their approach on four benchmark datasets:

  • CUB (Caltech-UCSD Birds): A fine-grained dataset.
  • FLO (Oxford Flowers): A fine-grained dataset.
  • AWA1 (Animals with Attributes 1): A coarse-grained dataset.
  • AWA2 (Animals with Attributes 2): A coarse-grained dataset.

They also mention using features from a ResNet101 backbone and features from a method proposed by [18] for comparison.

Key Insights and Ideas

  • Joint Training is Crucial: Augmenting a generative model with a jointly trained inference network significantly improves the ability to capture data distribution modes and generate better features.
  • Adversarial Joint Maximization for Coupling: A dedicated adversarial loss for joint maximization is more effective at enforcing visual-semantic coupling than simpler L2-norm based losses.
  • Inference Network for Discriminative Features: The latent representations from the inference network are valuable for the final recognition task, providing a boost in performance beyond just using synthesized features.
  • Wasserstein Alignment for Semantic Preservation: Using Wasserstein distance for semantic alignment is a novel and effective way to maintain semantic integrity, especially beneficial for fine-grained datasets.
  • Unifying Paradigms: TACO successfully integrates generative and representation learning approaches into a single, cohesive framework for GZSL.

Key Results

The proposed TACO method achieves state-of-the-art performance on all four benchmark datasets (CUB, FLO, AWA1, AWA2) for Generalized Zero-Shot Learning. Specifically, they report:

  • TACO consistently outperforms existing non-generative and generative GZSL approaches.
  • The method shows strong performance across both fine-grained (CUB, FLO) and coarse-grained (AWA1, AWA2) datasets.
  • Using a different feature extractor backbone (from [18]) still yields significant improvements, demonstrating the model's generalizability. For instance, on CUB, performance increased from 59.4% to 65.35%, and on AWA2, from 66.0% to 68.5%.
  • Ablation studies confirm the effectiveness of each component: the inference module, joint maximization, and Wasserstein alignment all contribute to performance gains.

Limitations and Future Work

  • Hyperparameter Sensitivity: The paper notes that hyperparameters like $\alpha_1$, $\alpha_2$, and $\gamma$ require careful tuning, with optimal values varying between fine-grained and coarse-grained datasets.
  • Future Work: The authors suggest exploring alternative methods for joint maximization and investigating alignments beyond Wasserstein alignment to further improve GZSL performance. They also mention the potential for future work to explore other ways of performing joint maximization.

Summary updated

Abstract

The performance of generative zero-shot methods mainly depends on the quality of generated features and how well the model facilitates knowledge transfer between visual and semantic domains. The quality of generated features is a direct consequence of the ability of the model to capture the several modes of the underlying data distribution. To address these issues, we propose a new two-level joint maximization idea to augment the generative network with an inference network during training which helps our model capture the several modes of the data and generate features that better represent the underlying data distribution. This provides strong cross-modal interaction for effective transfer of knowledge between visual and semantic domains. Furthermore, existing methods train the zero-shot classifier either on generated synthetic image features or latent embeddings produced by leveraging representation learning. In this work, we unify these paradigms into a single model which, in addition to synthesizing image features, also utilizes the representation learning capabilities of the inference network to provide discriminative features for the final zero-shot recognition task. We evaluate our approach on four benchmark datasets i.e. CUB, FLO, AWA1 and AWA2 against several state-of-the-art methods, and show its performance. We also perform ablation studies to analyze and understand our method more carefully for the Generalized Zero-shot Learning task.
BibTeX
@inproceedings{chandhok2021two,
  author = {Shivam Chandhok and Vineeth N. Balasubramanian},
  title = {{Two-Level Adversarial Visual-Semantic Coupling for Generalized Zero-shot Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  url = {https://openreview.net/forum?id=RRvJSGxlwC2J},
}
Figure