Zero-Shot Recognition via Optimal Transport

Wenlin Wang, Hongteng Xu, Guoyin Wang, Wenqi Wang, Lawrence CarinWACV 2021 · DOI

Introduction

Zero-Shot Learning (ZSL) aims to recognize object classes that were never seen during training by leveraging auxiliary information, such as attribute vectors (e.g., "has stripes," "is yellow"). Despite its potential, ZSL faces two primary challenges: Domain Shift and High-Bias. Domain shift occurs when the mapping from image features to attributes for seen classes fails to generalize to unseen classes due to differing high-dimensional distributions. High-bias is prevalent in Generalized ZSL (GZSL), where models naturally favor seen classes because they lack real training data for unseen ones.

Traditional methods often attempt point-by-point matching between samples and attributes, which is highly sensitive to noise. This paper proposes a shift from matching individual points to matching entire distributions using an Optimal Transport (OT) framework.

Methodology

The proposed OT-ZSL framework utilizes a generative approach to bridge the gap between seen and unseen classes.

Overview of the OT-ZSL framework showing training and testing flows.

Conditional Feature Generator

Instead of mapping images to attributes, the model employs a generator $g$ that synthesizes "fake" image features from attribute vectors and noise. This allows the model to "see" representations of unseen classes during training.

Optimal Transport (OT) Loss

To ensure the synthetic features are realistic, the model minimizes the Wasserstein distance (OT distance) between the distribution of generated features and real features extracted from a pretrained CNN. By solving the OT problem in its primal form, the authors avoid the stability issues common in Wasserstein GANs (WGANs), such as the need for strict 1-Lipschitz constraints.

IPOT Algorithm and Regularization

The Inexact Proximal point method for Optimal Transport (IPOT) is used to solve the OT problem efficiently and robustly. To maintain discriminative power and prevent "mode collapse," an attribute predictor $f$ is trained alongside the generator, forcing synthetic features to map accurately back to their original class attributes.

Datasets and Evaluation

The model was evaluated on several standard benchmarks, including AwA1 & AwA2 (Animals with Attributes), CUB (Caltech-UCSD Birds-200), SUN (Scene Recognition), and a large-scale test on ImageNet involving 1,000 seen and 360 unseen classes.

Key Results and Insights

OT-ZSL achieved state-of-the-art performance across multiple settings. In Generalized ZSL, it reached a Harmonic Mean ($H$) score of 69.6% on AwA1, significantly outperforming previous generative methods like f-CLSWGAN (59.6%). For standard ZSL, it achieved 74.3% accuracy on AwA1 and 68.1% on CUB.

t-SNE visualization showing that synthetic features for unseen classes cluster closely with real features.

The effectiveness of the generator is visually confirmed via t-SNE, where synthetic features for unseen classes (e.g., "horse," "sheep") cluster tightly with real, withheld features. By synthesizing data for unseen classes, the final softmax classifier is trained on a balanced dataset, drastically reducing bias toward seen classes. Furthermore, using cosine distance within the OT framework helped mitigate the "hubness" problem common in high-dimensional embeddings.

Conclusion

By framing Zero-Shot Learning as a distribution matching problem via Optimal Transport, this work effectively addresses domain shift and class bias. While the model can still struggle with "semantic confusion" between highly similar classes (e.g., blue whales vs. dolphins), the OT-ZSL framework provides a robust and scalable solution for recognizing the unseen.

Summary updated

BibTeX
@inproceedings{wang2021zero,
  author = {Wenlin Wang and Hongteng Xu and Guoyin Wang and Wenqi Wang and Lawrence Carin},
  title = {{Zero-Shot Recognition via Optimal Transport}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
  doi = {10.1109/wacv48630.2021.00351},
}
Figure