Learning Cross-domain Semantic-Visual Relation for Transductive Zero-Shot Learning

Jianyang Zhang, Fengmao Lv, Guowu Yang, Lei Feng, Yufeng Yu, Lixin Duan

Abstract

Zero-Shot Learning (ZSL) aims to recognize novel classes by leveraging semantic descriptions. A primary challenge in ZSL is the domain shift between seen and unseen classes. This paper introduces the Transferrable Semantic-Visual Network (TSVN), which reformulates ZSL as a domain adaptation problem. By shifting the objective from class prediction to relationship modeling and employing Domain-Specific Batch Normalization (DSBN), the proposed method aligns source and target distributions effectively. Experimental results across four benchmark datasets demonstrate state-of-the-art performance in the transductive ZSL setting.

Introduction

Zero-Shot Learning (ZSL) represents a significant frontier in computer vision, where models are tasked with identifying objects from categories not encountered during training. This is typically achieved by associating visual features with semantic attributes (e.g., "striped," "four-legged"). However, a persistent obstacle is the "domain shift" or "projection shift" problem: the mapping from visual features to semantic space learned on seen classes often fails to generalize to unseen classes.

While Domain Adaptation (DA) is a standard solution for distribution misalignment, it traditionally requires shared label spaces between the source and target domains. In ZSL, the label spaces are disjoint by definition, rendering conventional DA techniques inapplicable. This paper addresses this gap by redefining the learning objective to create a shared "relationship" space.

Methodology: The Transferrable Semantic-Visual Network (TSVN)

The authors propose the Transferrable Semantic-Visual Network (TSVN), which utilizes three core components to bridge the gap between seen and unseen domains.

1. Relationship Modeling

The fundamental innovation of TSVN is the transformation of ZSL from a multi-class classification problem into a binary relationship prediction task. Instead of predicting a specific class label, the model is fed pairs consisting of visual features and semantic attributes. The model then predicts whether the pair is "Similar" (matching) or "Dissimilar" (non-matching). Because the concept of "similarity" is universal across both seen and unseen classes, the label space becomes shared, enabling the application of domain adaptation.

2. Domain-Specific Batch Normalization (DSBN)

To align the distributions of the source (seen) and target (unseen) domains, the authors implement Domain-Specific Batch Normalization. Unlike standard alignment methods that might struggle with the extreme class imbalance inherent in relationship modeling (where dissimilar pairs vastly outnumber similar ones), DSBN employs separate batch normalization units for each domain at every layer. This allows the network to normalize domain-specific statistics independently while sharing the same transformation parameters, effectively "squashing" the two distributions together.

3. Attribute Reconstruction

To prevent the model from losing semantic integrity during the alignment process, an auxiliary decoder is integrated. This component forces the model to reconstruct the original semantic attributes from the internal latent representation, ensuring that the learned features remain grounded in the descriptive attributes of the classes.

Experimental Evaluation

The TSVN framework was evaluated in a transductive setting, where unlabeled images from the target classes are available during training. The model was tested on four major benchmarks:

  • aPY (attribute Pascal-Yahoo): 32 classes of diverse objects.
  • AwA (Animals with Attributes): 50 animal categories with 85-dimensional attributes.
  • CUB (Caltech-UCSD Birds): A fine-grained dataset of 200 bird species.
  • SUN (Scene UNderstanding): A massive dataset of 717 scene categories.

Key Results

The TSVN approach achieved state-of-the-art (SOTA) results across the board, particularly in the transductive setting:

  • AwA: Achieved a staggering 93.2% accuracy, significantly outperforming previous methods like ALE-T (60.6%).
  • aPY: Reached 53.3% on proposed splits, a notable jump from the baseline's 35.9%.
  • Distribution Alignment: Using t-SNE visualizations, the authors demonstrated that their DSBN method successfully aligned the source and target distributions, making them nearly indistinguishable to the metric network.

Discussion and Limitations

While TSVN sets new benchmarks, the authors note a practical trade-off in their implementation. Theoretically, the relationship between visual and semantic features is best modeled via a tensor product. However, due to the high dimensionality and computational cost of tensor products, the authors utilized vector concatenation. Future research may explore more efficient "fusion" methods to better capture these complex interactions. Additionally, while entropy regularization helped boost confidence in many cases, its performance on fine-grained datasets like CUB was less consistent, suggesting a need for more specialized regularization techniques in complex taxonomies.

Conclusion

By reframing Zero-Shot Learning as a relationship-based domain adaptation task, the TSVN framework effectively mitigates the projection shift problem. The use of Domain-Specific Batch Normalization provides a robust yet simple mechanism for distribution alignment, proving that even "mild" normalization strategies can yield superior results in the challenging transductive ZSL landscape.

Summary updated

Abstract

Highlights•We propose to address ZSL as a standard domain adaptation task.•A novel domain-specific batch normalization component is proposed to align domain gaps in semantic-visual fusions.•Extensive experiments show the effectiveness and robustness of our approach.
BibTeX
@article{zhang2023learning,
  author = {Jianyang Zhang and Fengmao Lv and Guowu Yang and Lei Feng and Yufeng Yu and Lixin Duan},
  title = {{Learning Cross-domain Semantic-Visual Relation for Transductive Zero-Shot Learning}},
  journal = {Pattern Recognition},
  year = {2023},
  url = {https://openreview.net/forum?id=S4WwvRnFgN},
}
Figure