Analyzing BERT Cross-lingual Transfer Capabilities in Continual Sequence Labeling

Juan Manuel Coria, Mathilde Veron, Sahar Ghannay, Guillaume Bernard, Hervé Bredin, Olivier Galibert, Sophie RossetCOLING 2022

TL;DR

This paper investigates how well BERT's cross-lingual transfer capabilities hold up in a continual learning setting, where models learn languages one by one. They find that while forward transfer (learning new languages) is robust, models tend to forget previously learned languages. However, this forgetting can be quickly recovered with minimal retraining, suggesting that knowledge is primarily stored in BERT's core layers.

What Problem Does This Paper Address?

Modern NLP models, especially large pre-trained ones like BERT, excel at transferring knowledge across languages. However, real-world deployment often involves scenarios where data becomes available sequentially, not all at once. This "continual learning" setting poses a challenge: how do models adapt to new languages without forgetting what they've already learned? Traditional methods often require retraining from scratch or maintaining many separate models, which is computationally expensive. This paper specifically addresses the question of how BERT's cross-lingual transfer abilities fare when languages are introduced sequentially, and whether the common problem of "catastrophic forgetting" hinders this transfer.

Methods

The researchers leverage multilingual BERT (specifically, the base model) for sequence labeling tasks. Their core methodology involves a continual learning setup where the model is trained on one language at a time, forming a "training sequence." They then evaluate performance on previously learned languages (backward transfer) and newly learned languages (forward transfer). To assess transfer, they compare performance against monolingual baselines (trained only on a single language) and a fully multilingual baseline (trained on all languages simultaneously). They also introduce metrics to quantify forward and backward transfer. A key aspect of their analysis is investigating the recovery of lost performance by retraining the model for a short period on a previously learned language.

Datasets

The paper uses two multilingual corpora for sequence labeling tasks:

  • MultiATIS++: A corpus for slot-filling tasks, derived from the ATIS corpus and translated into 9 languages (Spanish, Portuguese, German, French, Chinese, Japanese, Hindi, and Turkish).
  • MultiCoNER: A corpus for Named Entity Recognition (NER), containing annotations in 11 languages. For this study, they use 9 languages from this corpus (Bengali, German, English, Spanish, Hindi, Korean, Dutch, Turkish, and Chinese).

Key Insights and Ideas

  • Forward Transfer is Resilient, Forgetting is Real: While models can effectively learn new languages sequentially (forward transfer), they do exhibit forgetting of previously learned languages (backward transfer).
  • Continual Learning Metrics Can Be Misleading: Standard metrics that average performance across languages might not accurately reflect the distribution of transfer, especially when transfer is uneven across languages. The authors suggest that medians might be a better indicator.
  • Knowledge Resides in BERT's Core: The research strongly suggests that most of the knowledge retained from previous languages is stored in BERT's pre-trained encoder layers, rather than the task-specific classifier added for sequence labeling.
  • Rapid Recovery from Forgetting: Lost performance due to forgetting can be recovered remarkably quickly, often with just a single epoch of retraining. This is a significant finding for practical applications.
  • Progressive Shift Towards Multilingualism: As the model learns more languages sequentially, its parameters appear to shift towards a better multilingual initialization, which aids in learning new languages and recovering old ones.

Key Results

  • MultiATIS++:
    • Joint multilingual training generally outperforms monolingual training, with notable gains for low-resource languages like Hindi (+4.8%) and Turkish (+13.9%).
  • Continual training shows strong forward transfer, with performance on the last language in the sequence approaching multilingual levels.
  • Backward transfer (forgetting) is present, with performance on earlier languages dropping below monolingual baselines.
  • However, performance on the first language can be recovered and even improved upon with as little as one epoch of retraining, sometimes surpassing the 50-epoch multilingual baseline.
  • MultiCoNER:
    • Similar trends of forward transfer and forgetting are observed, though forgetting is more pronounced in this dataset.
  • Despite high forgetting, performance recovery with a single epoch of retraining is still significant, with some languages even matching or exceeding the multilingual topline.

Limitations and Future Work

  • Limitations:
    • The study focuses on two specific sequence labeling tasks (slot-filling and NER).
  • The "continual learning" setup assumes a fixed sequence of languages, and the order can impact results.
  • The paper notes that some MultiATIS++ translations might be unrealistic, potentially inflating transfer scores for certain languages.
  • Future Work:
    • Developing strategies to leverage fast recovery capabilities to reduce the cost of continual learning across languages.
  • Investigating the continual acquisition of languages not already present in multilingual BERT.
  • Exploring more robust continual learning metrics that better capture uneven transfer distributions.

Summary updated

BibTeX
@inproceedings{coria2022analyzing,
  author = {Juan Manuel Coria and Mathilde Veron and Sahar Ghannay and Guillaume Bernard and Hervé Bredin and Olivier Galibert and Sophie Rosset},
  title = {{Analyzing BERT Cross-lingual Transfer Capabilities in Continual Sequence Labeling}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure