TL-CL: Task And Language Incremental Continual Learning

Shrey Satapara, P. K. SrijithEMNLP 2024 · OpenReview

Abstract

In the evolving landscape of Natural Language Processing (NLP), multilingual models must frequently adapt to new tasks and languages post-deployment. This paper addresses the challenges of Task and Language Incremental Continual Learning (TLCL), where models learn sequences of task-language pairs. We propose Task and Language Specific Adapters (TLSA), a parameter-efficient approach that decouples task-specific logic from language-specific nuances. Our method achieves linear parameter growth relative to the number of tasks and languages, significantly mitigating catastrophic forgetting while maintaining high performance across diverse benchmarks.

Introduction

Continual Learning (CL) in NLP has traditionally been studied in two distinct paradigms: Task Incremental (TICL) and Language Incremental (LICL). TICL focuses on learning a sequence of different tasks in a fixed language, while LICL involves learning the same task across a sequence of new languages. However, real-world applications often require a hybrid approach where both tasks and languages are introduced incrementally.

Figure 1: Comparison of Task Incremental, Language Incremental, and the proposed Task & Language Incremental scenarios over time.

The primary challenge in this setting is catastrophic forgetting, where updates for new information overwrite weights critical for previous knowledge. Existing solutions often rely on task-language specific adapters, leading to a polynomial parameter explosion $O(|T| \times |L|)$. This paper introduces a framework to manage this complexity linearly.

Methodology: Task and Language Specific Adapters (TLSA)

To address the parameter explosion, we propose Task and Language Specific Adapters (TLSA). The core innovation lies in the "Segregation of Concerns," where task-specific knowledge (e.g., summarization logic) is decoupled from language-specific knowledge (e.g., Hindi syntax).

Architecture and Training

The model utilizes separate sets of adapters for tasks and languages. When encountering a specific task-language pair $(T_i, L_j)$, the corresponding task adapter and language adapter are activated.

Figure 2: (a) Integration of adapters within the Encoder-Decoder architecture. (b) Illustration of adapter activation and training transitions for different incremental scenarios.
  • Task-Specific Adapters: Shared across all languages for a given task.
  • Language-Specific Adapters: Shared across all tasks for a given language.

To prevent forgetting when adapters are reused across different pairs, we employ Elastic Weight Consolidation (EWC) as a regularizer. Additionally, we explore a Parameter Isolation variant (TLSA_PI) that saves snapshots of adapters to eliminate forgetting entirely, albeit at a higher storage cost.

Experimental Setup

We evaluated TLSA using four tasks (NLI, QA, Summarization, and Classification) across four languages (English, Spanish, Hindi, and Arabic). The benchmarks included XNLI, XQuAD, XLSum, and various sentiment analysis datasets. We defined three experimental scenarios:

  1. Complete: All tasks are learned in all languages.
  2. Partial: Tasks are available only in a subset of languages.
  3. Single Language Constraint: No repetition of tasks or languages in the sequence.

Key Results and Insights

TLSA demonstrates significant advantages in both performance and efficiency:

  • Parameter Efficiency: The method requires only 0.44% of the total model parameters to be trainable, making it highly scalable.
  • Performance: TLSA and TLSA_PI consistently outperformed established parameter-efficient methods like MAD-X and standard Adapter Tuning. In several benchmarks, TLSA performed comparably to Experience Replay (ER) without the need to store historical data, addressing privacy and storage constraints.
  • Knowledge Transfer: Decoupling allows for effective cross-task and cross-lingual transfer. For instance, language adapters placed in the early encoder layers and late decoder layers were found to be most effective for mapping to universal representations and adding language-specific generation bias, respectively.

Conclusion and Future Work

TLSA provides a scalable blueprint for building global-scale AI models that grow incrementally. While effective on mT5-small, future research should explore the scalability of this approach to massive LLMs and its performance on low-resource languages. Additionally, improving zero-shot generalization for unseen task-language combinations remains a critical area for development.

Summary updated

Abstract

This paper introduces and investigates the problem of Task and Language Incremental Continual Learning (TLCL), wherein a multilingual model is systematically updated to accommodate new tasks in previously learned languages or new languages for established tasks. This significant yet previously unexplored area holds substantial practical relevance as it mirrors the dynamic requirements of real-world applications. We benchmark a representative set of continual learning (CL) algorithms for TLCL. Furthermore, we propose Task and Language-Specific Adapters (TLSA), an adapter-based parameter-efficient fine-tuning strategy. TLSA facilitates cross-lingual and cross-task transfer and outperforms other parameter-efficient fine-tuning techniques. Crucially, TLSA reduces parameter growth stemming from saving adapters to linear complexity from polynomial complexity as it was with parameter isolation-based adapter tuning. We conducted experiments on several NLP tasks arising across several languages. We observed that TLSA outperforms all other parameter-efficient approaches without requiring access to historical data for replay.
BibTeX
@inproceedings{satapara2024tl,
  author = {Shrey Satapara and P. K. Srijith},
  title = {{TL-CL: Task And Language Incremental Continual Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=xFGy2e7hjc},
}
Figure