Learning to Solve NLP Tasks in an Incremental Number of Languages

Giuseppe Castellucci, Simone Filice, Danilo Croce, Roberto BasiliACL 2021

TL;DR

This paper tackles the challenge of incrementally adding new languages to existing NLP models without forgetting previously learned ones. They propose a "Teacher-Student" framework using Knowledge Distillation to transfer knowledge from an older model to a new one, enabling it to learn new languages while retaining performance on old ones. This approach significantly mitigates catastrophic forgetting, a major hurdle in continual learning for NLP.

What Problem Does This Paper Address?

In real-world NLP applications, models often need to support an ever-increasing number of languages. Traditionally, this involves retraining the model on data from all languages, which is expensive and time-consuming. Even worse, the original training data might become unavailable due to privacy concerns or storage limitations. A naive approach of fine-tuning only on new language data leads to "Catastrophic Forgetting" (CF), where the model loses its ability to perform well on previously learned languages. This paper addresses the critical need for a method that allows NLP models to learn new languages incrementally without sacrificing performance on existing ones, while also reducing the dependency on historical data.

Methods

The core of the proposed solution is a Continual Learning (CL) strategy based on Knowledge Distillation (KD), framed as a Teacher-Student setup.

  • Teacher-Student Framework: An existing, well-trained multilingual model (the "teacher") is used to guide a new, identical model (the "student").
  • Knowledge Distillation (KD): The teacher model provides "soft targets" (probability distributions over outputs) for unlabeled data from previously learned languages. The student model is trained to mimic these soft targets using a distillation loss ($L_{KD}$).
  • Task-Specific Loss ($L_T$): Simultaneously, the student model is trained on the new language data using the standard task-specific loss.
  • Combined Loss: The student's total loss is a combination of the task loss and the distillation loss ($L_{CL} = L_T + L_{KD}$). This dual objective ensures the student learns the new language while retaining knowledge from the old ones.

This approach differs from simple fine-tuning (which causes CF) and self-training (which can amplify errors) by leveraging the teacher's nuanced understanding of existing languages.

Datasets

The paper evaluates their method on three standard NLP tasks using the following datasets:

  • Sentence Classification: Multilingual Amazon Reviews Corpus (MARC) in 6 languages.
  • Sentence-Pairs Classification (Paraphrase Identification): PAWS-X dataset in 6 languages (Korean was excluded due to reproducibility issues).
  • Sequence Tagging (Named Entity Recognition): Merged CoNLL 2002 and 2003 datasets in 4 languages (English, Spanish, German, Dutch).

Key Insights and Ideas

  • Catastrophic Forgetting is a Real Problem: The experiments clearly show that simply fine-tuning on new languages leads to significant performance degradation on previously learned ones.
  • Knowledge Distillation is Key: The proposed CL-KD approach effectively transfers knowledge from the teacher model, preventing the student from forgetting past learning.
  • Teacher's Uncertainty is Valuable: Unlike self-training, which can amplify errors, using the teacher's soft targets preserves uncertainty and leads to more robust learning.
  • Order of Languages Matters Less Than Expected: The authors found that the order in which languages are introduced did not significantly impact the overall performance, even with linguistically diverse languages.
  • CL-KD Approaches Full Multilingual Performance: The CL-KD method achieves performance comparable to training a full multilingual model from scratch with all data, but with the significant advantage of incremental learning and reduced data requirements.

Key Results

The CL-KD strategy consistently outperformed other CL baselines (CL-Baseline, Self-Training, EWC) across all three tasks and datasets.

  • MARC (Sentiment Analysis): CL-KD reduced Mean Absolute Error (MAE) from 60.62 in the first step to 53.85 in the last step, outperforming CL-Baseline (57.30).
  • PAWS-X (Paraphrase Identification): CL-KD improved accuracy from 74.57 to 87.73, while CL-Baseline only reached 87.17.
  • CoNLL (NER): CL-KD achieved an F1 score of 79.80, compared to 72.67 for CL-Baseline.
  • Zero-Shot Performance: The gap between zero-shot performance (languages not yet trained on) and trained languages was substantial, highlighting the necessity of fine-tuning and the effectiveness of CL for bridging this gap.

Limitations and Future Work

  • Limitations:
    • The paper notes that the Self-Training method sometimes exhibited lower results, potentially due to error amplification.
  • The performance of EWC, a popular CL method, was not satisfactory in their specific NLP setting.
  • Reproducibility issues with the Korean language in the PAWS-X dataset led to its exclusion.
  • Future Work:
    • The authors suggest applying their methodology to other NLP tasks, such as Question Answering (QA).
  • Further investigation into the impact of language order and the potential for combining CL-KD with other pre-training objectives (like Masked Language Modeling) could be explored.

Summary updated

BibTeX
@inproceedings{castellucci2021learning,
  author = {Giuseppe Castellucci and Simone Filice and Danilo Croce and Roberto Basili},
  title = {{Learning to Solve NLP Tasks in an Incremental Number of Languages}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure