Learning One More Thing

Sebastian Thrun, Tom M. MitchellIJCAI 1995 · OpenReview

Abstract: Most research on machine learning has focused on scenarios in which a learner faces a single, isolated learning task. The lifelong learning framework assumes instead that the learner encounters a multitude of related learning tasks over its lifetime, providing the opportunity for the transfer of knowledge. This paper studies lifelong learning in the context of binary classification. It presents the invariance approach, in which knowledge is transferred via a learned model of the invariances of the domain. Results on learning to recognize objects from color images demonstrate superior generalization capabilities if invariances are learned and used to bias subsequent learning.

Introduction: Beyond Isolated Learning

Traditional supervised learning typically operates in a vacuum. When training a model to recognize a specific object, such as a "shoe," the process usually begins with a blank slate, relying solely on a provided set of positive and negative examples. This "isolated learning" paradigm ignores a crucial aspect of human intelligence: the ability to leverage a lifetime of experience to learn new concepts rapidly.

Thrun and Mitchell (1995) propose a shift toward lifelong learning. In this framework, a learner encounters a series of related tasks. By extracting general knowledge from "support sets"—data from previous tasks—the learner develops a bias that facilitates learning "one more thing," even when training data for the new task is extremely scarce.

The Invariance Approach

The core of the proposed method is the Invariance Approach. The goal is to learn what doesn't change across different tasks within a domain. For instance, in computer vision, an object's identity is generally invariant to small translations, rotations, or changes in lighting.

1. Learning Invariances

The system first trains an Invariance Network on pairs of images from support sets (e.g., bottles, hammers, books). This network learns to output a '1' if two images depict the same object and '0' otherwise. In doing so, it captures the underlying transformations that preserve object identity.

2. Transferring Knowledge via Slopes

When a new target task arrives (e.g., shoe vs. glasses), the learner uses the pre-trained Invariance Network to generate slopes (gradients). These slopes provide a "hint" about the local geometry of the feature space, indicating how the output should (or should not) change in response to input perturbations.

Standard backpropagation often fails to capture the true shape of the target function when data is sparse.

3. Tangent-Prop and EBNN

The model incorporates these slopes as additional constraints during training using a technique called Tangent-Prop. Instead of merely fitting the individual data points, the Explanation-Based Neural Network (EBNN) fits the shape of the function around those points. This biases the learner toward solutions that respect the learned domain invariances.

By incorporating learned slopes, the Invariance Approach constrains the hypothesis to match the local geometry of the domain.

Empirical Evaluation

The authors tested this approach on a dataset of 700 color images featuring seven distinct objects. The images were downsampled to 10x10 pixels to create 300-dimensional input vectors capturing color, brightness, and saturation.

The most significant results appeared in one-shot learning scenarios. When provided with only one positive and one negative example of a new object:

  • Standard Backpropagation: Achieved an accuracy of 59.1%.
  • Invariance Approach (EBNN): Achieved an accuracy of 74.8%.

Qualitatively, the EBNN-trained models demonstrated much higher robustness to object positioning. While standard models often fixated on irrelevant background features or specific locations, the lifelong learner correctly identified that position was an invariant property.

Limitations and Future Directions

While effective, the approach assumes that invariances are consistent across all tasks in a domain. If a new task requires a different set of invariances, the learned bias could potentially hinder performance. Additionally, the computational cost of training on image pairs scales quadratically with the number of examples, and capturing complex, non-linear invariances in domains like speech or financial prediction remains a challenge for future research.

Summary updated

Abstract

Most research on machine learning has focused on scenarios in which a learner faces a single, isolated learning task. The lifelong learning framework assumes instead that the learner encounters a multitude of related learning tasks over its lifetime, providing the opportunity for the transfer of knowledge. This paper studies lifelong learning in the context of binary classification. It presents the invariance approach, in which knowledge is transferred via a learned model of the invariances of the domain. Results on learning to recognize objects from color images demonstrate superior generalization capabilities if invariances are learned and used to bias subsequent learning.
BibTeX
@inproceedings{thrun1995learning,
  author = {Sebastian Thrun and Tom M. Mitchell},
  title = {{Learning One More Thing}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {1995},
  url = {https://openreview.net/forum?id=SJbfGNG_WS},
}
Figure