Towards Robust and Efficient Continual Language Learning

Adam Fisch, Amal Rannen-Triki, Razvan Pascanu, Jörg Bornschein, Angeliki Lazaridou, Elena Gribovskaya, Marc'Aurelio RanzatoarXiv 2023 · OpenReview

TL;DR

This paper tackles the challenge of efficiently adapting large language models (LLMs) to new tasks by exploring continual learning. They introduce a novel benchmark to systematically evaluate how well LLMs can leverage knowledge from previous tasks, even when faced with potentially detrimental "negative transfer," and propose a simple yet effective checkpoint selection strategy to improve this process.

What Problem Does This Paper Address?

Large language models (LLMs) have become incredibly powerful, but adapting them to new, specific tasks often involves fine-tuning. While fine-tuning is more efficient than training from scratch, doing it independently for each new task leads to accumulating computational costs and misses opportunities to leverage knowledge gained from previous tasks. This is the domain of continual learning. The core problem is that simply fine-tuning on a sequence of tasks can lead to "negative transfer," where knowledge from a previous task actually hinders performance on a new one. The authors aim to develop methods that can efficiently adapt LLMs to new tasks while being robust to these negative transfer effects, essentially asking: "How can we intelligently reuse past learning to learn new things faster and better?"

Methods

The paper proposes a selective sequential fine-tuning approach. Instead of always starting fine-tuning from the original, massive pre-trained model (the "default"), their method intelligently selects a previously fine-tuned model (a "checkpoint") as the starting point for a new task. This selection is guided by a gradient boosted decision tree (GBDT) trained on "meta-features" derived from pairs of tasks. These features capture aspects like relative performance (0-shot and 5-shot), weight changes during fine-tuning, and task metadata (like task family). The GBDT acts as a discriminator to identify "positive transfer" candidates. If no confident positive transfer candidate is found, it defaults to the original pre-trained model. This is a departure from "naïve" sequential fine-tuning, which simply uses the most recently trained model.

Datasets

The authors leverage a large collection of 55 popular and publicly available English language tasks, primarily drawing from the FLAN dataset (Wei et al., 2022), but excluding translation tasks. They organize these tasks into approximate "families" for analysis. A key contribution is the creation of a new benchmark of task sequences designed to specifically test different transfer scenarios: high positive transfer, high negative transfer, no expected effect, or mixtures thereof. These sequences are constructed based on extensive pairwise transfer analysis of the 55 tasks.

Key Insights and Ideas

  • The transfer landscape is complex: Pairwise analysis reveals that while some tasks benefit greatly from prior fine-tuning, others are actively harmed. This highlights the need for more sophisticated transfer strategies than simple sequential fine-tuning.
  • "Naïve" sequential fine-tuning is brittle: Simply using the most recently trained model as a starting point can be highly sensitive to task ordering, leading to significant performance degradation when negative transfer is present.
  • Intelligent checkpoint selection matters: By learning to identify which past checkpoints are likely to facilitate positive transfer (and crucially, which to avoid), significant improvements in learning efficiency and robustness can be achieved.
  • Not all transfer is equal: The authors emphasize the importance of distinguishing between positive and negative transfer, and developing methods that can exploit the former while mitigating the latter.

Key Results

The paper presents results on various task sequence types. For sequences with intended positive transfer (e.g., A → C and B → C are positive), their selective method achieves performance close to an "oracle" (which knows the best path in hindsight), significantly outperforming the "naïve" sequential fine-tuning. For instance, in a positive-positive sequence, their method achieved a median relative PerfAUC of 15.1, compared to 16.9 for naïve sequential fine-tuning and 19.9 for the oracle.

Crucially, for sequences involving negative transfer, their selective approach demonstrates its robustness. When faced with negative intermediate tasks, their method successfully defaults to the pre-trained model, achieving a relative PerfAUC of 0 (meaning performance equivalent to independent fine-tuning), whereas naïve sequential fine-tuning can result in significant negative PerfAUC scores (e.g., -8.34 in one case).

Limitations and Future Work

The authors acknowledge several limitations:

  • Focus on T5 models and parameter initialization: Their work is specific to T5 models and only explores transfer via parameter initialization, not other forms of efficiency like parameter reduction.
  • Computational cost of feature extraction: Computing the features for their checkpoint selector involves training and evaluating many models, which doesn't scale well with a very large number of tasks.
  • Potential for false positives: As the number of tasks grows, the selective classifier might incorrectly identify poor checkpoints.
  • Stochasticity: Noise from SGD and batch selection can introduce confounding effects.

Future work could explore other efficiency mechanisms (e.g., parameter sharing), develop more scalable feature extraction methods, and investigate techniques for robust checkpoint selection in the presence of many tasks. They also hope their benchmark will spur further research into analyzing and building robust continual learning algorithms.

Summary updated

Abstract

As the application space of language models continues to evolve, a natural question to ask is how we can quickly adapt models to new tasks. We approach this classic question from a continual learning perspective, in which we aim to continue fine-tuning models trained on past tasks on new tasks, with the goal of "transferring" relevant knowledge. However, this strategy also runs the risk of doing more harm than good, i.e., negative transfer. In this paper, we construct a new benchmark of task sequences that target different possible transfer scenarios one might face, such as a sequence of tasks with high potential of positive transfer, high potential for negative transfer, no expected effect, or a mixture of each. An ideal learner should be able to maximally exploit information from all tasks that have any potential for positive transfer, while also avoiding the negative effects of any distracting tasks that may confuse it. We then propose a simple, yet effective, learner that satisfies many of our desiderata simply by leveraging a selective strategy for initializing new models from past task checkpoints. Still, limitations remain, and we hope this benchmark can help the community to further build and analyze such learners.
BibTeX
@misc{fisch2023towards,
  author = {Adam Fisch and Amal Rannen-Triki and Razvan Pascanu and Jörg Bornschein and Angeliki Lazaridou and Elena Gribovskaya and Marc'Aurelio Ranzato},
  title = {{Towards Robust and Efficient Continual Language Learning}},
  year = {2023},
  eprint = {2307.05741},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2307.05741},
}
Figure