Lifelong Learning Without a Task Oracle

Amanda Rios, Laurent Itti

Abstract

Continual learning in deep neural networks often relies on a "task oracle" to provide task identities at inference time, a requirement that limits real-world applicability. This paper proposes the use of lightweight task-assigning mappers to infer task identities autonomously. By utilizing fixed feature embeddings and efficient mapping strategies, the proposed system achieves near-oracle performance with minimal memory overhead, bridging the gap between task-dependent models and autonomous lifelong learning.

Introduction

A primary challenge in deep learning is "catastrophic forgetting," where a model loses previously acquired knowledge upon learning new tasks. While "task-dependent" architectures mitigate this by partitioning network weights or using task-specific parameters, they typically require a Task Oracle to specify the current task during testing. In practical scenarios, such as autonomous robotics, such an oracle is unavailable. This research addresses this limitation by introducing Task Mappers—modules designed to identify task IDs on the fly, enabling models to function independently of external task labels.

Proposed Methodology

The authors implement a two-stage pipeline consisting of a Task Mapper and a Fine-Grained Classifier. The Task Mapper analyzes input features to predict a task ID, which then directs the Fine-Grained Classifier (utilizing Parameter Superposition or PSP) to activate the appropriate sub-network.

Architecture of the proposed system where a Task Mapper substitutes the traditional Task Oracle.

Task Mapper Variants

Several mapping strategies were evaluated to determine the most effective balance of accuracy and efficiency:

  1. Incremental Unsupervised Mappers: Techniques such as Nearest Means (NMC) and Gaussian Mixture Models (GMMC) store task prototypes and assign inputs based on proximity in feature space.
  2. Supervised Prototype Mapping (ARTMAP): An incremental neural network architecture designed for category learning without forgetting.
  3. Perceptron with Coreset Replay (PCR): A shallow network trained for task identification, utilizing a small "coreset" buffer to maintain performance over time.
Overview of various task mapper architectures evaluated, including unsupervised and supervised approaches.

A key finding was the importance of using fixed feature embeddings from a pre-trained backbone (e.g., ImageNet-pretrained ResNet). Stable representations proved superior to evolving, task-specific features for the purpose of task identification.

Experimental Evaluation

The system was tested across three distinct lifelong learning paradigms:

  • Inter-Dataset Learning (8-Datasets): Sequential learning of diverse datasets (e.g., Flowers, Birds, Cars), testing cross-domain discrimination.
  • Permuted MNIST: 25 tasks involving randomly shuffled pixels of MNIST digits.
  • Intra-Dataset Learning (CIFAR-100 Super-Classes): 10 tasks where each task represents a visually similar super-class, representing the most challenging scenario for task mapping.

Key Results and Insights

  • High Accuracy: The GMMC mapper achieved 93.9% task identification accuracy on the 8-dataset benchmark, nearly matching oracle performance.
  • Efficiency: The mappers are highly lightweight, adding only approximately 1.7% to the total parameter count. For instance, the GMMC mapper required only 82 KB of memory for prototypes.
  • Performance Gains: The Mapper + PSP-BD pipeline significantly outperformed task-independent baselines like EWC or GEM, which do not utilize task IDs.
  • Domain Difficulty: Task mapping is significantly more difficult for intra-dataset tasks (e.g., distinguishing between different insect types) compared to inter-dataset tasks (e.g., distinguishing a bird from a car).

Conclusion and Future Work

This work demonstrates that autonomous task identification is a viable path toward oracle-free lifelong learning. While the mappers excel in inter-dataset scenarios, a performance gap remains in fine-grained intra-dataset tasks. Future research will focus on developing more generalizable visual embeddings that can evolve during the lifelong learning process without sacrificing the stability required for accurate task mapping.

Summary updated

Abstract

Supervised deep neural networks are known to undergo a sharp decline in the accuracy of older tasks when new tasks are learned, termed “catastrophic forgetting”. Many state-of-the-art solutions to continual learning rely on biasing and/or partitioning a model to accommodate successive tasks incrementally. However, these methods largely depend on the availability of a task-oracle to confer task identities to each test sample, without which the models are entirely unable to perform. To address this shortcoming, we propose and compare several candidate task-assigning mappers which require very little memory overhead: (1) Incremental unsupervised prototype assignment using either nearest means, Gaussian Mixture Models or fuzzy ART backbones; (2) Supervised incremental prototype assignment with fast fuzzy ARTMAP; (3) Shallow perceptron trained via a dynamic coreset. Our proposed model variants are trained either from pre-trained feature extractors or task-dependent feature embeddings of the main classifier network. We apply these pipeline variants to continual learning benchmarks, comprised of either sequences of several datasets or within one single dataset. Overall, these methods, despite their simplicity and compactness, perform very close to a ground truth oracle, especially in experiments of inter-dataset task assignment. Moreover, best-performing variants only impose an average cost of 1.7% parameter memory increase.
BibTeX
@inproceedings{rios2020lifelong,
  author = {Amanda Rios and Laurent Itti},
  title = {{Lifelong Learning Without a Task Oracle}},
  booktitle = {ICTAI},
  year = {2020},
  url = {https://openreview.net/forum?id=cMbk0qzwpqz},
}
Figure