Polyglot Contextual Representations Improve Crosslingual Transfer

Phoebe Mulcaire, Jungo Kasai, Noah A. Smith

Abstract

Phoebe Mulcaire, Jungo Kasai, Noah A. Smith. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers).

2019.

Introduction

In natural language processing (NLP), crosslingual transfer leverages data from resource-rich languages like English to improve models for lower-resource languages. While traditional methods relied on static multilingual word embeddings, the advent of Contextual Word Representations (CWRs) such as ELMo and BERT has shifted the paradigm. This paper introduces Rosita, a polyglot extension of the ELMo architecture trained on multiple languages simultaneously. The researchers investigate whether training a single model on dissimilar language pairs (e.g., English/Arabic or English/Chinese) can overcome potential "interference" from conflicting syntactic structures to provide superior crosslingual transfer.

Methods

Rosita extends the ELMo framework by training a single BiLSTM language model on a concatenated corpus of two languages. The authors evaluate three primary configurations:

  1. MONOCHAR: A standard monolingual ELMo-style model using character-level CNNs.
  2. ROSITACHAR: A polyglot model trained on a target language plus English using character-level inputs. This model relies on implicit alignment of shared features like punctuation or numbers.
  3. ROSITAWORD: A polyglot model combining character-level CNNs with word-type embeddings (fastText) aligned via the MUSE method, providing explicit crosslingual anchors.

Pretraining utilized approximately 60 million tokens per language from diverse sources (e.g., Billion Word Benchmark for English, Xinhua for Chinese). Evaluation focused on Universal Dependency (UD) parsing, Semantic Role Labeling (SRL), and Named Entity Recognition (NER) using the Ontonotes 5.0 dataset for consistent crosslingual annotation.

Results and Analysis

The polyglot approach consistently outperformed monolingual and non-contextual baselines across all tasks.

Dependency Parsing

Rosita models achieved state-of-the-art results in dependency parsing, surpassing the winning systems from the CoNLL 2018 shared task. As shown in Figure 1, RositaChar (ARA+ENG) reached a 78.02 LAS, outperforming the top-ranked Stanford system.

Comparison of RositaChar against CoNLL 2018 winners on dependency parsing.

SRL and NER Performance

The benefits of polyglot training were particularly pronounced in Semantic Role Labeling and Named Entity Recognition. In Chinese SRL, Rosita models achieved an $F_1$ score of 74.84, a significant improvement over the previous best of 62.83. Similarly, polyglot models showed consistent gains in NER, with the Arabic model reaching 76.19 $F_1$ compared to the monolingual 75.02.

Comprehensive results across UD Parsing, SRL, and NER tasks.

Key Insights

  • Implicit Alignment: ROSITACHAR often matched or exceeded the performance of ROSITAWORD, suggesting that CWRs can learn crosslingual connections from unlabeled data without explicit bilingual dictionaries.
  • Regularization: Training on a second language appears to act as a powerful regularizer, helping the model learn more robust and generalizable linguistic structures even across dissimilar scripts.
  • Contextual Superiority: Contextual representations proved far more effective for transfer than static embeddings. Even when static alignment marginally degraded performance, the polyglot CWRs provided a net gain.

Conclusion

Rosita demonstrates that polyglot contextual representations significantly improve crosslingual transfer for diverse NLP tasks. While the study focused on language pairs, it highlights the potential for scaling polyglot training to more languages. Future work may explore these benefits within the BERT framework and address sensitivities to gold-standard data size in lower-resource settings.

Summary updated

BibTeX
@inproceedings{mulcaire2019polyglot,
  author = {Phoebe Mulcaire and Jungo Kasai and Noah A. Smith},
  title = {{Polyglot Contextual Representations Improve Crosslingual Transfer}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=rkEeTz-_Wr},
}
Figure