TL;DR
LAMOL (LAnguage MOdeling for Lifelong language learning) is a simple yet powerful method that enables a single language model to learn a sequence of diverse NLP tasks without forgetting previous ones. By leveraging the generative nature of language models to "replay" pseudo-samples of past tasks, LAMOL prevents catastrophic forgetting without requiring extra memory or increasing model capacity.
What Problem Does This Paper Address?
The "holy grail" of artificial general intelligence is lifelong learning: the ability of a machine to learn a stream of tasks sequentially, accumulating knowledge over time. However, standard neural networks suffer from catastrophic forgetting. When trained on a new task (Task B), the model's weights shift to optimize for the new data distribution, often completely erasing its ability to perform a previous task (Task A).
Existing solutions usually fall into three categories:
- Regularization: Constraining weight updates (e.g., EWC), which often limits the model's ability to learn new tasks.
- Architecture: Adding new parameters for every task, which leads to unsustainable model growth.
- Data-based: Storing a subset of real data from old tasks to "rehearse." This raises significant privacy concerns and requires substantial memory.
LAMOL addresses these gaps by using pseudo-rehearsal: instead of storing real data, the model learns to generate its own training samples for previous tasks.
Methods
The core innovation of LAMOL is treating a single Language Model (LM) as both the solver and the data generator.
1. Unified QA Format
Following the "decaNLP" approach, the authors frame every NLP task (sentiment analysis, semantic parsing, etc.) as a Question Answering (QA) problem. A sample consists of a Context, a Question, and an Answer.
2. Dual-Purpose Training
The model (specifically a pre-trained GPT-2) is trained with two simultaneous objectives:
- QA Objective: Given a context and question, generate the correct answer.
- LM Objective: Given a special generation token (
GEN), generate the entire sequence (Context + Question + Answer).
3. Pseudo-Rehearsal
When the model moves to Task $N$, it first uses its current weights to generate a set of "pseudo-samples" representing Tasks $1$ through $N-1$. It then trains on a mixture of these generated samples and the new real data from Task $N$. This "reminds" the model of what it previously knew.
4. Task-Specific Tokens
To prevent the distribution of older tasks from "vanishing" as the task list grows, the authors introduced task-specific tokens (e.g., [SQuAD] instead of a generic [GEN]). This allows the model to explicitly target and balance the generation of samples for every previous task.
Datasets
The authors evaluated LAMOL on nine diverse NLP benchmarks:
- decaNLP Tasks: SQuAD (Question Answering), WikiSQL (Semantic Parsing), SST (Sentiment Analysis), QA-SRL (Semantic Role Labeling), and WOZ (Goal-Oriented Dialogue).
- Text Classification: AGNews, Amazon, DBPedia, Yahoo, and Yelp.
Key Insights and Ideas
- The "Free" Generator: Most pseudo-rehearsal methods require a separate GAN or VAE to generate old data. LAMOL realizes that a Language Model is already a generator. By adding a simple generation token, you get a data simulator for "free."
- QA as a Universal Interface: By framing everything as QA, the model doesn't need task-specific "heads" or output layers. This makes the transition between tasks seamless.
- Robustness to "Chaos": Interestingly, the authors noted that the generated pseudo-samples aren't always perfect—sometimes they are "chaotic" or nonsensical. However, even imperfect data serves as a sufficient anchor to prevent the model's weights from drifting too far, effectively acting as a data-driven regularizer.
- Efficiency: Because it doesn't store real data or grow the architecture, LAMOL is highly practical for deployment in memory-constrained or privacy-sensitive environments.
Key Results
- Near-Upper Bound Performance: LAMOL achieved results only 2–3% lower than multitasking (where the model learns all tasks simultaneously with all data available), which is generally considered the theoretical ceiling for lifelong learning.
- Outperforming Baselines: LAMOL significantly outperformed traditional LLL methods like EWC (Elastic Weight Consolidation) and GEM (Gradient Episodic Memory). For example, on a sequence of three tasks, LAMOL maintained an average score of ~79.7%, while EWC dropped to ~47.0%.
- Stability: With task-specific tokens, the model showed virtually no sign of "intransigence" (inability to learn new tasks) or "forgetting" even as the sequence of tasks grew.
Limitations and Future Work
- Generation Quality vs. Context Length: The authors noted that as contexts get longer (like in SQuAD), it becomes harder for the model to generate high-quality pseudo-samples. This "chaos" in generation is a bottleneck for more complex tasks.
- Computational Overhead: While LAMOL saves memory, it increases training time because the model must generate pseudo-samples at the start of every new task training phase.
- Task Ordering: The paper explores task order (e.g., large-to-small vs. small-to-large) and finds it matters, but a definitive strategy for optimal ordering in a truly "streaming" environment remains an open question.
Future work could involve improving the quality of the generated pseudo-samples or finding ways to make the generation process more efficient during the training loop.
Summary updated