Visual-semantic dual-disentangling for generalized zero-shot learning

Ayou Han, Guan Yang, Xiaoming Liu, Yang LiuJournal of Image and Graphics 2023 · DOI

This academic paper, published in the Journal of Image and Graphics (2023), presents a novel approach to one of the trickiest problems in computer vision: recognizing things you’ve never seen before.

TL;DR

The authors propose VSD-GZSL, a framework that improves Generalized Zero-Shot Learning by "untangling" visual features and semantic attributes. By stripping away irrelevant noise from both images and text descriptions, the model creates a cleaner, more accurate mapping that significantly boosts classification performance for both seen and unseen categories.

What Problem Does This Paper Address?

In Zero-Shot Learning (ZSL), a model is trained on "seen" classes (e.g., horses and tigers) and must then identify "unseen" classes (e.g., zebras) using only text-based attributes (e.g., "has stripes," "looks like a horse"). Generalized ZSL (GZSL) is even harder because, at test time, the model must distinguish between all classes simultaneously.

The core challenge is the visual-semantic gap. Raw visual features extracted by deep networks (like ResNet) contain "noise" irrelevant to the labels (like background or lighting). Conversely, human-defined semantic attributes often include details that aren't easily captured in a single photo. Most existing models try to force these two messy signals together, leading to a "bias" where the model over-identifies everything as a seen class.

Methods

The authors introduce a Visual-Semantic Dual-Disentangling framework. Instead of treating features as monolithic blocks, they split them apart:

  1. Feature Generation: They use a conditional Variational Autoencoder (cVAE) to synthesize visual features for unseen classes, providing the model with "hallucinated" training data.
  2. Visual Disentanglement: This module splits visual features into two parts: semantic-consistent (features that actually match the description) and semantic-irrelevant (noise like background).
  3. Semantic Disentanglement: Similarly, it splits text attributes into feature-relevant and feature-irrelevant components.
  4. Total Correlation (TC) Penalty: To ensure these splits are clean, they use a TC penalty—a technique from information theory—to force the disentangled components to be statistically independent.
  5. Cross-Modal Reconstruction: To prove the disentangling worked, the model tries to reconstruct the semantic attributes using only the "consistent" visual parts, and vice versa.
  6. Relation Network: A sub-network scores the compatibility between the cleaned-up visual and semantic features to ensure they truly align.

Datasets

The model was evaluated on four gold-standard GZSL benchmarks:

  • AwA2 (Animals with Attributes 2): 50 animal classes with 85 attributes.
  • CUB (Caltech-UCSD Birds): 200 bird species with 312 fine-grained attributes.
  • SUN (Scene Attribute): 717 scene categories with 102 attributes.
  • FLO (Oxford Flowers): 102 flower types with 1,024-dimensional descriptions.

Key Insights and Ideas

  • The "Dual" is Crucial: Most previous research only tried to "clean" the visual side. This paper argues that the semantic side (the text attributes) is also noisy and needs disentangling to find the "feature-relevant" core.
  • Independence via TC: Simply splitting a vector into two doesn't mean the information is separated. By using a Total Correlation penalty and a discriminator, the authors force the "noise" and "signal" into truly separate channels.
  • Mutual Consistency: The "Aha!" moment is that by training a classifier only on the mutually consistent parts of the image and text, the model becomes much more robust to the "seen class bias" that plagues GZSL.

Key Results

The authors used the Harmonic Mean (H)—a metric that balances accuracy on both seen and unseen classes—to prove their point. VSD-GZSL outperformed the baseline (SDGZSL) and several state-of-the-art methods:

  • SUN Dataset: Saw the biggest jump, with the Harmonic Mean increasing by 6.2%.
  • CUB Dataset: Improved by 3.2%.
  • AwA2 Dataset: Improved by 1.6%.
  • FLO Dataset: Improved by 1.5%.

The model proved particularly effective at identifying "unseen" classes, which is usually the weakest link in GZSL systems.

Limitations and Future Work

While the results are impressive, the authors acknowledge a few areas for growth:

  • Backbone Dependency: The performance is still heavily tied to the quality of the pre-trained visual backbone (e.g., ResNet vs. ViT). As shown in their ablation study, switching to a Vision Transformer (ViT) significantly boosts results, suggesting the "raw" features still matter immensely.
  • Manual Attributes: The system relies on human-annotated attributes, which are expensive to create.
  • Future Direction: The authors suggest moving toward attention mechanisms at the image level (rather than just feature vectors) and using Transformers to discover multi-modal relationships automatically without relying so heavily on manual attribute lists.

Summary updated

Abstract

目的传统的零样本学习(zero-shot learning,ZSL)旨在依据可见类别的数据和相关辅助信息对未见类别的数据进行预测分类,而广义零样本学习(generalized zero-shot learning,GZSL)中分类的类别既可能属于可见类也可能属于不可见类,这更符合现实的应用场景。基于生成模型的广义零样本学习的原始特征和生成特征不一定编码共享属性所指的语义相关信息,这样会导致模型倾向于可见类,并且分类时忽略了语义信息中与特征相关的有用信息。为了分解出相关的视觉特征和语义信息,提出了视觉—语义双重解纠缠框架。方法首先,使用条件变分自编码器为不可见类生成视觉特征,再通过一个特征解纠缠模块将其分解为语义一致性和语义无关特征。然后,设计了一个语义解纠缠模块将语义信息分解为特征相关和特征无关的语义。其中,利用总相关惩罚来保证分解出来的两个分量之间的独立性,特征解纠缠模块通过关系网络来衡量分解的语义一致性,语义解纠缠模块通过跨模态交叉重构来保证分解的特征相关性。最后,使用两个解纠缠模块分离出来的语义一致性特征和特征相关语义信息联合学习一个广义零样本学习分类器。结果实验在4个广义零样本学习公开数据集AWA2(animals with attributes2)、CUB(caltech-ucsd birds-200-2011)、SUN(SUN attribute)和FLO(Oxford flowers)上取得了比Baseline更好的结果,调和平均值在AwA2、CUB、SUN和FLO上分别提升了1.6%、3.2%、6.2%和1.5%。结论在广义零样本学习分类中,本文提出的视觉—语义双重解纠缠方法经实验证明比基准方法取得了更好的性能,并且优于大多现有的相关方法。
BibTeX
@article{han2023visual,
  author = {Ayou Han and Guan Yang and Xiaoming Liu and Yang Liu},
  title = {{Visual-semantic dual-disentangling for generalized zero-shot learning}},
  journal = {Journal of Image and Graphics},
  year = {2023},
  doi = {10.11834/jig.220486},
}
Figure