Representation as a Service

Ouais Alsharif, Philip Bachman, Joëlle Pineau

Abstract

Consider a Machine Learning Service Provider (MLSP) designed to rapidly create highly accurate learners for a never-ending stream of new tasks. The challenge is to produce task-specific learners that can be trained from few labeled samples, even if tasks are not uniquely identified, and the number of tasks and input dimensionality are large. In this paper, we argue that the MLSP should exploit knowledge from previous tasks to build a good representation of the environment it is in, and more precisely, that useful representations for such a service are ones that minimize generalization error for a new hypothesis trained on a new task. We formalize this intuition with a novel method that minimizes an empirical proxy of the intra-task small-sample generalization error. We present several empirical results showing state-of-the art performance on single-task transfer, multitask learning, and the full lifelong learning problem.

Introduction

The modern machine learning landscape often requires adapting to a continuous stream of novel tasks with minimal labeled data. We conceptualize this as a Machine Learning Service Provider (MLSP) scenario. For instance, a service might need to deploy custom face recognition models for numerous clients, each providing only a handful of examples. Traditional deep learning approaches struggle here due to three primary hurdles: the small-data problem (overfitting), lack of task correspondence (inconsistent labels across tasks), and scalability (the inability to store all historical data or retrain globally for every new task).

While techniques like unsupervised pre-training offer some relief, they often fail to leverage the discriminative information present in labeled data from related tasks. We propose LeaDR (Learning Discriminative Representations), a framework that treats the ability to generalize from small samples as the primary training objective for a shared representation.

The LeaDR Framework

LeaDR decomposes the learning architecture into two components: a shared feature extractor ($f_\theta$), typically a deep neural network, and simple, fast-to-train task-specific learners ($h_i$), such as logistic regression.

The core innovation of LeaDR is its "proactive" meta-learning objective. Instead of optimizing the representation to minimize training error on the current task, it optimizes for small-sample generalization. For each incoming task, the algorithm performs a pseudo-train/validate split:

  1. It partitions a small batch of samples into a "pseudo-train" set and a "pseudo-validation" set.
  2. A temporary learner is trained on the pseudo-train set using the current representation.
  3. The shared representation parameters ($\theta$) are updated by calculating the gradient of the loss on the pseudo-validation set.

This process essentially tunes the feature extractor to ensure that a simple learner, when trained on very few samples, achieves high performance on unseen data within that task.

The LeaDR algorithm steps for updating representations based on small-sample generalization.

Experimental Evaluation

We evaluated LeaDR across three learning regimes: single-task transfer, standard multitask learning, and lifelong learning.

Single-Task Transfer and Multitask Learning

Using CIFAR-100 to pre-train a representation, we tested on the NIPS 2011 Transfer Learning Challenge. LeaDR achieved an 8% accuracy improvement over standard CNNs in "one-shot" scenarios (one sample per class). It also outperformed S3C (Spike-and-Slab Sparse Coding), a leading unsupervised method, while requiring significantly fewer labeled samples. In standard multitask settings using the Landmine and London Schools datasets, LeaDR demonstrated robust performance across varying task complexities.

Lifelong Learning and Scalability

In lifelong learning simulations with 1,000 sequential tasks (using 20 Newsgroups and high-dimensional CIFAR-100), LeaDR consistently outperformed the Efficient Lifelong Learning Algorithm (ELLA). Notably, while ELLA's $O(d^3)$ complexity makes it difficult to scale to high-dimensional inputs, LeaDR scales linearly with input dimensionality and maintains high accuracy even as the number of samples per task remains low.

Discussion and Conclusion

LeaDR offers several advantages for lifelong learning systems. By optimizing for generalization directly, it creates a representation that is inherently "ready-to-use" for new, low-data tasks. Its task-agnostic nature solves the correspondence problem, as updates are driven by local validation loss rather than global label consistency. Furthermore, the algorithm is $O(1)$ in memory and computation per task, making it ideal for streaming environments.

Future work includes addressing the computational approximations used in the meta-update and extending the framework to non-parametric learners like K-Nearest Neighbors. While LeaDR assumes a degree of environmental homogeneity, its ability to build "Representation as a Service" marks a significant step toward scalable, data-efficient lifelong learning.

Summary updated

Abstract

Consider a Machine Learning Service Provider (MLSP) designed to rapidly create highly accurate learners for a never-ending stream of new tasks. The challenge is to produce task-specific learners that can be trained from few labeled samples, even if tasks are not uniquely identified, and the number of tasks and input dimensionality are large. In this paper, we argue that the MLSP should exploit knowledge from previous tasks to build a good representation of the environment it is in, and more precisely, that useful representations for such a service are ones that minimize generalization error for a new hypothesis trained on a new task. We formalize this intuition with a novel method that minimizes an empirical proxy of the intra-task small-sample generalization error. We present several empirical results showing state-of-the art performance on single-task transfer, multitask learning, and the full lifelong learning problem.
BibTeX
@misc{alsharif2014representation,
  author = {Ouais Alsharif and Philip Bachman and Joëlle Pineau},
  title = {{Representation as a Service}},
  year = {2014},
  eprint = {1404.4108},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1404.4108},
}
Figure