Zero-shot Learning with Semantic Output Codes

Mark Palatucci, Dean Pomerleau, Geoffrey E. Hinton, Tom M. MitchellNeurIPS 2009

TL;DR

This paper introduces a formal framework for Zero-Shot Learning (ZSL), allowing classifiers to recognize novel categories that were entirely absent from the training set. By mapping raw inputs to a high-dimensional "semantic feature space" (Semantic Output Codes), the authors demonstrate that a model can successfully decode words a person is thinking about from fMRI brain scans, even for words the model has never seen before.

What Problem Does This Paper Address?

Traditional machine learning follows a "closed-world" assumption: a classifier can only predict labels it has seen during training. However, in many real-world domains—like computer vision or neural decoding—the number of possible classes is massive, and collecting labeled examples for every single one is intractable.

For instance, if you want to build a "thought-to-text" brain-computer interface, you cannot possibly record fMRI data for every noun in the English language. This paper tackles the challenge of extrapolation: how can a system learn the underlying "building blocks" of a class so that it can recognize a brand-new class by simply knowing its semantic description?

Methods

The authors propose the Semantic Output Code (SOC) classifier, which decomposes the classification task into two distinct stages:

  1. The Mapping Function ($S$): Instead of mapping raw input ($X$) directly to a label ($Y$), the model learns to map $X$ to a $p$-dimensional Semantic Feature Space. For example, instead of learning "this brain activity = dog," it learns "this brain activity = has fur, has four legs, is a carnivore."
  2. The Knowledge Base ($L$): This is a pre-defined library that contains the semantic encodings for a vast number of classes (including those not in the training set). To classify a new input, the model predicts the semantic features and then finds the closest match in the Knowledge Base using a distance metric (like 1-Nearest Neighbor).

Theoretical Contribution: The paper provides a PAC-style (Probably Approximately Correct) theoretical analysis. They derive a formal guarantee (error bounds) that determines how many training examples are needed to ensure the semantic prediction is accurate enough to successfully recover the correct novel class label.

Datasets

The authors evaluate their approach on a neural decoding task using:

  • fMRI Dataset: Brain activity patterns from nine participants viewing 60 different concrete nouns (e.g., bear, train, screwdriver).
  • Semantic Knowledge Bases:
    • corpus5000: Automated features derived from co-occurrence statistics in the Google Trillion-Word-Corpus.
  • human218: 218 semantic properties (e.g., "is it manmade?", "can you hold it?") labeled by humans via Amazon Mechanical Turk.
  • noun940: A larger set of 940 English nouns used to test the model's ability to pick the correct word out of a large "dictionary" of possibilities.

Key Insights and Ideas

  • Semantic Building Blocks: Just as speech recognition uses a small set of phonemes to recognize an infinite variety of words, ZSL uses semantic properties to recognize an infinite variety of objects.
Predicted vs. target semantic features for 'bear' and 'dog'.
  • Human Intuition Wins: Features derived from human labeling (human218) significantly outperformed automated corpus statistics. This suggests that the way our brains encode meaning is more closely aligned with functional/descriptive properties (size, usage, category) than simple word-association frequency.
  • Error Tolerance: The theoretical analysis shows that the model doesn't need to be perfect at predicting every semantic feature. As long as the "total error" in the semantic space is less than half the distance to the next closest class in the Knowledge Base, the model will still pick the right label.

Key Results

  • Binary Discrimination: In a "leave-two-out" test (where the model had to distinguish between two words it had never seen), the SOC classifier achieved 80.9% accuracy using human-labeled features.
  • Large-Scale Decoding: When asked to identify a "hidden" word from a pool of 941 candidates, the model's performance was remarkably high. For many participants, the median rank accuracy was above 90%, meaning the correct (unseen) word was usually in the top 10% of the model's predictions.
Rank accuracy of the zero-shot classifier on different word sets and semantic feature bases.
  • Concrete Success: For some subjects, the model correctly picked the exact word (e.g., bear) out of 941 alternatives more than 10% of the time—thousands of times better than chance (0.1%).

Limitations and Future Work

  • Category Bias: The authors noted that certain categories (like animals or vehicles) were much easier to decode than others (like furniture or insects). This suggests the current semantic features might not capture the nuances of all object classes equally well.
  • Knowledge Base Dependency: The system's "imagination" is limited by its Knowledge Base. If a class isn't in the semantic library, the model can never predict it.
  • Linearity: The current model uses linear regression for the mapping function. Future work could explore non-linear mappings (like deep neural networks) to capture more complex relationships between brain activity and semantics.
  • Scaling: While 1,000 words is impressive, scaling this to the full complexity of human language (abstract concepts, verbs, etc.) remains a significant hurdle.

Summary updated

BibTeX
@inproceedings{palatucci2009zero,
  author = {Mark Palatucci and Dean Pomerleau and Geoffrey E. Hinton and Tom M. Mitchell},
  title = {{Zero-shot Learning with Semantic Output Codes}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2009},
}
Figure