Semantically Grounded Visual Embeddings for Zero-Shot Learning

Shah Nawaz, Jacopo Cavazza, Alessio Del Bue

Abstract

Zero-shot learning methods rely on fixed visual and semantic embeddings, extracted from independent vision and language models, both pre-trained for other large-scale tasks. This is a weakness of current zero-shot learning frameworks as such disjoint embeddings fail to adequately associate visual and textual information to their shared semantic content. Therefore, we propose to learn semantically grounded and enriched visual information by computing a joint image and text model with a two-stream network on a proxy task. To improve this alignment between image and textual representations, provided by attributes, we leverage ancillary captions to provide grounded semantic information. Our method, dubbed joint embeddings for zero-shot learning is evaluated on several benchmark datasets, improving the performance of existing state-of-the-art methods in both standard (+1.6% on aPY, +2.6% on FLO) and generalized (+2.1% on AWA2, +2.2% on CUB) zero-shot recognition.

Introduction: The Disjoint Embedding Problem

Zero-Shot Learning (ZSL) aims to classify images into categories never seen during training by mapping visual features to semantic descriptions. Traditionally, this relies on two independent components: visual embeddings from models like ResNet and semantic embeddings from attribute lists (e.g., "has spots").

However, these "disjoint" embeddings often fail to capture the subtle nuances required for fine-grained recognition. Standard visual features are not trained to recognize the specific semantic details described in attributes. For instance, while multiple species might share the attribute "spotted," the specific visual manifestation of those spots differs significantly across classes.

Comparison of attributes vs. captions for fine-grained bird species, highlighting the semantic nuances captured by natural language.

Methodology: Joint Embeddings for ZSL (JE-ZSL)

To address this gap, the proposed JE-ZSL framework pre-trains visual embeddings to be "semantically grounded." Instead of using off-the-shelf features, the model learns a shared space through a two-stream architecture:

  • Two-Stream Network: A visual module (ResNet-101) and a semantic module (BERT) process image-caption pairs.
  • Structure-Preserving Alignment Loss: A triplet-style loss ensures that images are closer to their corresponding captions than to others. Neighborhood constraints are applied to ensure that semantically similar images and sentences remain clustered together in the embedding space.
  • Proxy Task Training: The model is first trained on large-scale image-caption datasets like MSCOCO and Conceptual Captions. The resulting "grounded" visual stream is then used to extract features for ZSL tasks.

Experimental Results

The researchers integrated JE-ZSL into established ZSL backbones, including CDL, f-CLSWGAN, and tf-VAEGAN. The grounded embeddings consistently outperformed standard ResNet features across various benchmarks:

  • Standard ZSL: Significant gains were observed on the FLO (+2.6%) and aPY (+1.6%) datasets.
  • Generalized ZSL (GZSL): In the more challenging GZSL setting, where the test set includes both seen and unseen classes, the method improved performance on AWA2 by +2.1% and CUB by +2.2%.
  • The Power of Captions: When using captions specifically tailored to the ZSL datasets (such as CUB), the performance boost was even more pronounced, yielding a +7.9% increase in Top-1 accuracy for the f-CLSWGAN backbone.

Conclusion

JE-ZSL demonstrates that bridging the gap between vision and language at the feature extraction level—rather than just at the final classification stage—is critical for robust Zero-Shot Learning. By leveraging natural language captions to ground visual features, the model captures semantic nuances that sparse attributes miss, particularly in fine-grained recognition tasks. Future work may involve incorporating even broader semantic sources, such as knowledge graphs, to further enrich these visual representations.

Summary updated

Abstract

Zero-shot learning methods rely on fixed visual and semantic embeddings, extracted from independent vision and language models, both pre-trained for other large-scale tasks. This is a weakness of current zero-shot learning frameworks as such disjoint embeddings fail to adequately associate visual and textual information to their shared semantic content. Therefore, we propose to learn semantically grounded and enriched visual information by computing a joint image and text model with a two-stream network on a proxy task. To improve this alignment between image and textual representations, provided by attributes, we leverage ancillary captions to provide grounded semantic information. Our method, dubbed joint embeddings for zero-shot learning is evaluated on several benchmark datasets, improving the performance of existing state-of-the-art methods in both standard (+1.6% on aPY, +2.6% on FLO) and generalized (+2.1% on AWA2, +2.2% on CUB) zero-shot recognition.
BibTeX
@inproceedings{nawaz2022semantically,
  author = {Shah Nawaz and Jacopo Cavazza and Alessio Del Bue},
  title = {{Semantically Grounded Visual Embeddings for Zero-Shot Learning}},
  booktitle = {Workshops @ CVPR},
  year = {2022},
  url = {https://openreview.net/forum?id=c9HwK2tF3u},
}
Figure