Zero-data Learning of New Tasks

Hugo Larochelle, Dumitru Erhan, Yoshua Bengio

Introduction

In modern machine learning, models typically require thousands of examples to recognize a specific class. However, humans can often identify new objects based solely on a description. This paper by Larochelle et al. (2008) introduces Zero-data Learning (now more commonly known as Zero-Shot Learning), exploring how models can generalize to entirely new tasks or classes using high-level descriptions instead of labeled examples.

This approach addresses three critical challenges: the Scale Problem (the impossibility of labeling all 30,000+ object categories), the Cold-Start Problem (predicting for new users or biological targets without prior data), and Dynamic Environments (adapting to new symbols or classes via specification rather than retraining).

Methods

The authors formalize zero-data learning by leveraging auxiliary "task descriptions" $d(z)$ to perform a task $z$ on input $x$. They propose two primary frameworks for this interaction:

The Input Space View

This approach treats the task description as an additional feature vector. The input $x$ and description $d(z)$ are concatenated and fed into a standard classifier, such as an SVM. The model essentially treats the description as just another set of features.

The Model Space View

The core innovation involves using the description to define the parameters of the model itself—a "model of models." The task description $d(z)$ determines the weights of the classifier $f_z(x)$. Specific implementations include:

  • LinGen: A generative model where class means are linear functions of their descriptions.
  • LinDisc: A discriminative model where decision boundaries are derived from descriptions.
  • NNet-0-1: A non-linear version using a neural network to learn hidden representations before applying description-based logic.
Character recognition task setup with 7x5 pixel task descriptions.

Experimental Setup

The theories were tested across two distinct domains:

  1. Character Recognition: Using 40 license plate character classes and 36 handwritten alphanumeric classes. Descriptions were 7x5 "canonical" pixel maps representing the ideal version of each character.
  2. Drug Discovery: Predicting molecular activity against new biological targets using 25-dimensional vectors describing "binding pockets."

Results and Discussion

The experiments confirmed that generalization to unseen tasks is achievable because different tasks share underlying components. For instance, learning how specific pixel patterns constitute a "vertical line" across known characters allows a model to identify a new character described with that trait.

Test error rates on unseen characters as a function of the number of training classes.

Key findings include:

  • Generalization Trends: In character recognition, test error on unseen classes decreased significantly as the number of training classes increased. On license plate data, the best models reached nearly perfect generalization once they had seen enough other character types.
  • Model Superiority: The Model Space View outperformed the Input Space View by preventing the model from ignoring descriptions in favor of simply memorizing training classes.
  • Class-Level Overfitting: The authors observed that models can become so confident in training classes that they effectively "blind" themselves to new classes, even when the input clearly matches the new description.

Conclusion

This work laid the foundation for zero-shot learning, proving that machines can perform tasks they have never seen if provided with adequate "instructions." While the system is dependent on description quality, it provides a powerful mechanism for a "warm start" in fields like drug discovery and dynamic character recognition.

Summary updated

Abstract

We introduce the problem of zero-data learning, where a model must generalize to classes or tasks for which no training data are available and only a description of the classes or tasks are provided. Zero-data learning is useful for problems where the set of classes to distinguish or tasks to solve is very large and is not entirely covered by the training data. The main contributions of this work lie in the presentation of a general formalization of zero-data learning, in an experimental analysis of its properties and in empirical evidence showing that generalization is possible and significant in this context. The experimental work of this paper addresses two classification problems of character recognition and a multitask ranking problem in the context of drug discovery. Finally, we conclude by discussing how this new framework could lead to a novel perspective on how to extend machine learning towards AI, where an agent can be given a specification for a learning problem before attempting to solve it (with very few or even zero examples).
BibTeX
@inproceedings{larochelle2008zero,
  author = {Hugo Larochelle and Dumitru Erhan and Yoshua Bengio},
  title = {{Zero-data Learning of New Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
}
Figure