Zero-Shot Learning with Structured Embeddings.

Zeynep Akata, Honglak Lee, Bernt Schiele

TL;DR

This paper introduces a Structured Joint Embedding (SJE) framework for fine-grained image classification, enabling zero-shot learning (classifying images from categories not seen during training). They demonstrate that unsupervised embeddings derived from text and hierarchies can achieve competitive, and sometimes superior, results to supervised methods, especially when combined.

What Problem Does This Paper Address?

Fine-grained image classification, where categories are visually very similar (e.g., different bird species), is challenging due to the high cost of annotating large datasets with specific labels. Traditional supervised learning requires many labeled examples per class. Zero-shot learning aims to overcome this by leveraging auxiliary information to classify unseen classes. This paper specifically tackles the extreme case of zero-shot learning where no labeled data is available for some target classes, and explores how to effectively use various forms of "side information" about classes to bridge the gap between visual features and class labels.

Methods

The core contribution is the Structured Joint Embedding (SJE) framework. SJE learns a compatibility function that measures how well an image's features (input embedding) align with a class's side information (output embedding). For zero-shot classification, an image is assigned to the class whose output embedding yields the highest compatibility score.

Key aspects of their method include:

  • Input Embeddings: They utilize state-of-the-art image features, specifically Fisher Vectors (FV) and deep convolutional neural network (CNN) features (from AlexNet and GoogLeNet).
  • Output Embeddings: They explore various sources of class-specific information:
  • Human-Annotated Attributes: Binary or continuous values describing object characteristics (e.g., "has wings," "color is red").
  • Unsupervised Text Embeddings: Word vectors learned from large text corpora like Wikipedia (Word2Vec, GloVe) and Bag-of-Words representations.
  • Hierarchical Embeddings: Semantic similarity measures derived from the WordNet ontology.
  • Weakly-supervised Word2Vec: A novel variant that fine-tunes Word2Vec embeddings using a specialized, fine-grained text corpus.
  • Parameter Learning: They use a multiclass objective, similar to structured SVMs, to learn the compatibility function's parameters. This involves optimizing a matrix that maps input embedding dimensions to output embedding dimensions.
  • Combining Embeddings: A crucial aspect is their exploration of combining multiple output embeddings (e.g., text + hierarchy) through concatenation or a weighted ensemble, showing that complementary information can significantly boost performance.

Datasets

The paper evaluates their methods on three datasets:

  • Animals with Attributes (AWA): A standard benchmark for attribute-based zero-shot learning.
  • Caltech-UCSD Birds (CUB): A fine-grained dataset focusing on bird species.
  • Stanford Dogs (Dogs): A fine-grained dataset of dog breeds, which they note is the first attempt at zero-shot learning on this dataset.

They use standard train/validation/test splits, ensuring that test classes are disjoint from training classes for the zero-shot setting.

Key Insights and Ideas

  • Unsupervised Embeddings Can Be Powerful: The most striking finding is that purely unsupervised embeddings (learned from Wikipedia) can achieve competitive, and sometimes superior, results to supervised attribute-based methods, especially on fine-grained datasets. This suggests that large, readily available text corpora contain rich semantic information about object classes.
  • Complementarity is Key: Combining different types of output embeddings (e.g., text-based and hierarchical) consistently improves performance. This highlights that each embedding type captures different facets of class semantics, and their fusion leads to a more robust representation.
  • Fine-grained Text Matters: Using a specialized text corpus (e.g., bird-specific text for bird classification) for learning or fine-tuning embeddings significantly boosts performance compared to using general-purpose text. This indicates that domain-specific language captures more relevant nuances for fine-grained tasks.
  • Continuous Attributes Outperform Binary: Representing attributes as continuous values (confidence levels) rather than simple binary presence/absence leads to substantial improvements, especially when combined with deep image features.
  • SJE is a Flexible Framework: The SJE framework is general and can accommodate various input and output embedding types, making it a versatile tool for multimodal learning problems.

Key Results

  • State-of-the-Art Improvements: They achieve substantially improved state-of-the-art results on AWA and CUB datasets.
  • On AWA, they reach 73.9% accuracy with combined unsupervised embeddings, significantly outperforming the previous supervised SoA of 48.5%.
  • On CUB, they achieve 51.7% accuracy with combined embeddings, surpassing the previous supervised SoA of 26.9%.
  • Unsupervised SoA: Their best unsupervised results on AWA (60.1%) and CUB (29.9%) outperform previous supervised SoA results.
  • Dogs Dataset Performance: They report 35.1% accuracy on the Dogs dataset using unsupervised embeddings, a notable achievement for this challenging fine-grained task.
  • Qualitative Success: Visualizations show that different embeddings capture distinct semantic aspects of classes (e.g., social behavior vs. habitat), and their combination synthesizes these aspects effectively.

Limitations and Future Work

  • Gap with Human Attributes: Despite impressive gains, there remains a significant gap between unsupervised embeddings and human-annotated attributes on AWA and CUB. This suggests that current text-based methods still need improvement to fully capture the discriminative power of fine-grained human annotations.
  • Future Work: The authors plan to explore other methods for combining multiple output embeddings and to enhance the discriminative power of unsupervised and weakly-supervised embeddings for fine-grained classification. They also suggest investigating other sources of semantic information.

Summary updated

Abstract

Image classification has advanced significantly in recent years with the availability of large-scale image sets. However, fine-grained classification remains a major challenge due to the annotation cost of large numbers of fine-grained categories. This project shows that compelling classification performance can be achieved on such categories even without labeled training data. Given image and class embeddings, we learn a compatibility function such that matching embeddings are assigned a higher score than mismatching ones; zero-shot classification of an image proceeds by finding the label yielding the highest joint compatibility score. We use state-of-the-art image features and focus on different supervised attributes and unsupervised output embeddings either derived from hierarchies or learned from unlabeled text corpora. We establish a substantially improved state-of-the-art on the Animals with Attributes and Caltech-UCSD Birds datasets. Most encouragingly, we demonstrate that purely unsupervised output embeddings (learned from Wikipedia and improved with fine-grained text) achieve compelling results, even outperforming the previous supervised state-of-the-art. By combining different output embeddings, we further improve results.
BibTeX
@inproceedings{akata2015zero,
  author = {Zeynep Akata and Honglak Lee and Bernt Schiele},
  title = {{Zero-Shot Learning with Structured Embeddings.}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2015},
  url = {https://openreview.net/forum?id=H1W-r0Z_bS},
}
Figure