LAMOL: LAnguage MOdeling for Lifelong Language Learning

Fan-Keng Sun, Cheng-Hao Ho, Hung-Yi Lee

TL;DR

LAMOL (LAnguage MOdeling for Lifelong language learning) is a simple yet powerful method that enables a single language model to learn a sequence of diverse NLP tasks without forgetting previous ones. By leveraging the generative nature of language models to "replay" pseudo-samples of past tasks, LAMOL prevents catastrophic forgetting without requiring extra memory or increasing model capacity.

What Problem Does This Paper Address?

The "holy grail" of artificial general intelligence is lifelong learning: the ability of a machine to learn a stream of tasks sequentially, accumulating knowledge over time. However, standard neural networks suffer from catastrophic forgetting. When trained on a new task (Task B), the model's weights shift to optimize for the new data distribution, often completely erasing its ability to perform a previous task (Task A).

Existing solutions usually fall into three categories:

  1. Regularization: Constraining weight updates (e.g., EWC), which often limits the model's ability to learn new tasks.
  2. Architecture: Adding new parameters for every task, which leads to unsustainable model growth.
  3. Data-based: Storing a subset of real data from old tasks to "rehearse." This raises significant privacy concerns and requires substantial memory.

LAMOL addresses these gaps by using pseudo-rehearsal: instead of storing real data, the model learns to generate its own training samples for previous tasks.

Methods

The core innovation of LAMOL is treating a single Language Model (LM) as both the solver and the data generator.

LAMOL Overview: A single model acts as both learner and generator.

1. Unified QA Format

Following the "decaNLP" approach, the authors frame every NLP task (sentiment analysis, semantic parsing, etc.) as a Question Answering (QA) problem. A sample consists of a Context, a Question, and an Answer.

2. Dual-Purpose Training

The model (specifically a pre-trained GPT-2) is trained with two simultaneous objectives:

  • QA Objective: Given a context and question, generate the correct answer.
  • LM Objective: Given a special generation token (GEN), generate the entire sequence (Context + Question + Answer).
Dual-Purpose Training Objectives

3. Pseudo-Rehearsal

When the model moves to Task $N$, it first uses its current weights to generate a set of "pseudo-samples" representing Tasks $1$ through $N-1$. It then trains on a mixture of these generated samples and the new real data from Task $N$. This "reminds" the model of what it previously knew.

4. Task-Specific Tokens

To prevent the distribution of older tasks from "vanishing" as the task list grows, the authors introduced task-specific tokens (e.g., [SQuAD] instead of a generic [GEN]). This allows the model to explicitly target and balance the generation of samples for every previous task.

Datasets

The authors evaluated LAMOL on nine diverse NLP benchmarks:

  • decaNLP Tasks: SQuAD (Question Answering), WikiSQL (Semantic Parsing), SST (Sentiment Analysis), QA-SRL (Semantic Role Labeling), and WOZ (Goal-Oriented Dialogue).
  • Text Classification: AGNews, Amazon, DBPedia, Yahoo, and Yelp.

Key Insights and Ideas

  • The "Free" Generator: Most pseudo-rehearsal methods require a separate GAN or VAE to generate old data. LAMOL realizes that a Language Model is already a generator. By adding a simple generation token, you get a data simulator for "free."
  • QA as a Universal Interface: By framing everything as QA, the model doesn't need task-specific "heads" or output layers. This makes the transition between tasks seamless.
  • Robustness to "Chaos": Interestingly, the authors noted that the generated pseudo-samples aren't always perfect—sometimes they are "chaotic" or nonsensical. However, even imperfect data serves as a sufficient anchor to prevent the model's weights from drifting too far, effectively acting as a data-driven regularizer.
  • Efficiency: Because it doesn't store real data or grow the architecture, LAMOL is highly practical for deployment in memory-constrained or privacy-sensitive environments.

Key Results

  • Near-Upper Bound Performance: LAMOL achieved results only 2–3% lower than multitasking (where the model learns all tasks simultaneously with all data available), which is generally considered the theoretical ceiling for lifelong learning.
  • Outperforming Baselines: LAMOL significantly outperformed traditional LLL methods like EWC (Elastic Weight Consolidation) and GEM (Gradient Episodic Memory). For example, on a sequence of three tasks, LAMOL maintained an average score of ~79.7%, while EWC dropped to ~47.0%.
  • Stability: With task-specific tokens, the model showed virtually no sign of "intransigence" (inability to learn new tasks) or "forgetting" even as the sequence of tasks grew.

Limitations and Future Work

  • Generation Quality vs. Context Length: The authors noted that as contexts get longer (like in SQuAD), it becomes harder for the model to generate high-quality pseudo-samples. This "chaos" in generation is a bottleneck for more complex tasks.
  • Computational Overhead: While LAMOL saves memory, it increases training time because the model must generate pseudo-samples at the start of every new task training phase.
  • Task Ordering: The paper explores task order (e.g., large-to-small vs. small-to-large) and finds it matters, but a definitive strategy for optimal ordering in a truly "streaming" environment remains an open question.

Future work could involve improving the quality of the generated pseudo-samples or finding ways to make the generation process more efficient during the training loop.

Summary updated

Abstract

Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/jojotenya/LAMOL.
BibTeX
@inproceedings{sun2020lamol,
  author = {Fan-Keng Sun and Cheng-Hao Ho and Hung-Yi Lee},
  title = {{LAMOL: LAnguage MOdeling for Lifelong Language Learning}},
  booktitle = {International Conference on Learning Representations},
  year = {2020},
  url = {https://openreview.net/forum?id=PxD6JH09Eme},
}
Figure