Lifelong Learning of Discriminative Representations

Ouais Alsharif, Philip Bachman, Joelle Pineau

Introduction

In modern machine learning, the abundance of massive, well-labeled datasets is often taken for granted. However, real-world scenarios frequently involve "Machine Learning Service Providers" (MLSPs) who must handle a continuous stream of diverse tasks, each accompanied by only a few labeled examples. To address this, Alsharif et al. propose LeaDR (Learning Discriminative Representations), a framework designed to build a shared feature extractor optimized for efficient learning on new, unseen tasks with minimal data.

Problem Statement: The MLSP Challenge

The authors focus on the challenge faced by an MLSP, such as a company providing facial recognition services to multiple clients. Each client has unique classes (different people) and very limited data. Standard deep learning approaches fail in this context due to:

  1. Small Sample Sizes: Deep models typically require thousands of labels to avoid overfitting.
  2. Task Correspondence: Labels are often inconsistent across different tasks (e.g., "Class 1" for Client A is not "Class 1" for Client B), preventing direct label sharing.
  3. Scalability: Lifelong learning requires handling a task stream without storing all historical data or retraining from scratch.

While Multitask Learning (MTL) assumes a fixed task set, and previous lifelong learning methods like ELLA struggle with high-dimensional inputs, LeaDR aims for a scalable, representation-centric solution.

The LeaDR Algorithm

The core of LeaDR is training a shared representation ($f_\theta$) to minimize the generalization error of future learners. Instead of optimizing for a specific set of labels, the model learns to be a "good servant" for task-specific heads ($h_i$).

The LeaDR algorithmic flow: from task arrival to representation update via pseudo-train/validate splits.

The process involves seven key steps as illustrated above:

  1. Task Arrival: A new task $T_i$ arrives with $m$ samples.
  2. Data Split: Data is split into a training set ($tr_i^n$) and a validation set ($va_i^{m-n}$).
  3. Feature Extraction: Training data is passed through the shared representation $f_\theta$.
  4. Hypothesis Generation: A simple algorithm $\mathcal{A}_i$ (e.g., logistic regression) is applied to the features to produce a task-specific hypothesis $h_i$.
  5. Minibatch Sampling: A minibatch $\tilde{va}_i^n$ is sampled from the validation set.
  6. Error Computation: The error of $h_i$ is computed on the validation minibatch.
  7. Representation Update: The parameters $\theta$ of the shared representation are updated to minimize this validation error, while keeping $h_i$ fixed.

This "meta-learning" approach explicitly minimizes an empirical proxy of intra-task generalization error, ensuring the learned features are inherently discriminative for new classes.

Experimental Evaluation

The authors evaluated LeaDR across three regimes: Single-Task Transfer (CIFAR-100), Multitask Learning (Landmine and London Schools datasets), and large-scale Lifelong Learning (CIFAR-100 and 20-Newsgroups).

Key Results

  • State-of-the-Art Transfer: On the NIPS 2011 challenge, LeaDR outperformed Spike-and-Slab Sparse Coding (S3C) and standard ConvNets, achieving superior accuracy with only half the labeled samples.
  • Scalability in Lifelong Learning: LeaDR significantly outperformed ELLA, particularly on high-dimensional raw pixel data, showing lower variance and higher accuracy as samples per task increased.
  • Few-Shot Performance: In one-shot scenarios (1 sample per class), LeaDR provided an 8% accuracy gain over standard convolutional networks on CIFAR-100.

Discussion and Future Work

LeaDR demonstrates that utility-driven representation learning is highly effective for lifelong learning. Its $O(1)$ complexity relative to the number of tasks makes it truly scalable. However, some limitations remain. The current implementation uses a first-order approximation for speed; second-order gradients might improve performance. Additionally, while the model assumes tasks come from a related domain, extreme heterogeneity might lead to "catastrophic forgetting." Future work could explore non-parametric learners (like K-Nearest Neighbors) as task-specific heads.

Summary updated

Abstract

Consider a Machine Learning Service Provider (MLSP) designed to rapidly create highly accurate learners for a never-ending stream of new tasks. The challenge is to produce task-specific learners that can be trained from few labeled samples, even if tasks are not uniquely identified, and the number of tasks and input dimensionality are large. In this paper, we argue that the MLSP should exploit knowledge from previous tasks to build a good representation of the environment it is in, and more precisely, that useful representations for such a service are ones that minimize generalization error for a new hypothesis trained on a new task. We formalize this intuition with a novel method that minimizes an empirical proxy of the intra-task small-sample generalization error. We present several empirical results showing state-of-the art performance on single-task transfer, multitask learning, and the full lifelong learning problem.
BibTeX
@misc{alsharif2014lifelong,
  author = {Ouais Alsharif and Philip Bachman and Joelle Pineau},
  title = {{Lifelong Learning of Discriminative Representations}},
  year = {2014},
  eprint = {1404.4108},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1404.4108},
}
Figure