A Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings

Niels van der Heijden, Samira Abnar, Ekaterina Shutova

Abstract

The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-of-the-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multiple baselines and show that it performs at or above state-of-the-art level in zero-shot transfer settings. Finally, we show that our method allows for better knowledge sharing across languages in a joint training setting.

Introduction

Most state-of-the-art NLP models require vast amounts of annotated data, which is unavailable for the majority of the world's languages. To bridge this "data gap," researchers utilize multilingual embeddings—mathematical representations of words in a shared vector space where semantically similar words across languages (e.g., "apple" and "manzana") are positioned closely together.

While static embeddings (one vector per word) were previously the standard, contextualized embeddings—where vectors change based on surrounding words—have significantly improved performance. However, much of this research has focused on single languages or sentence-level tasks. This paper investigates which architectures and pretraining methods are most effective for word-level tasks, such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging, particularly when transferring knowledge from high-resource languages like English to low-resource targets.

Methodology

The authors evaluate several major architectures, focusing on how different pretraining objectives influence cross-lingual transferability.

Transformer-based Models

The study examines Multilingual BERT (mBERT) and the Cross-lingual Language Model (XLM). These models utilize the Transformer architecture and are pretrained on massive Wikipedia datasets. mBERT primarily uses Masked Language Modeling (MLM), while XLM introduces Translation Language Modeling (TLM) to leverage parallel corpora for better alignment.

Transformer-based pretraining objectives: Masked Language Modeling (MLM) and Translation Language Modeling (TLM).

LASER-based Models (The Proposed Approach)

LASER (Language-Agnostic SEntence Representations) is a 5-layer BiLSTM encoder originally designed to translate sentences into a language-agnostic space. The authors propose extracting word-level information from this sentence encoder using two methods:

  • LASER-top: Utilizing the hidden states from the final layer of the encoder to represent sub-words.
  • LASER-elmo: Applying a weighted average of all five layers (inspired by the ELMo architecture) to capture both syntactic and semantic information.
The LASER encoder architecture, a multi-layer BiLSTM used to generate sentence and word-level embeddings from BPE tokens.

Experimental Setup

The models were evaluated on the CoNLL-2002 and CoNLL-2003 shared tasks. The primary evaluation metric was zero-shot transfer: training a model on English data and testing it on target languages (Dutch, German, Spanish, Hungarian, and Basque) without any target-language training labels. Hungarian and Basque were specifically chosen to test model robustness on languages linguistically distant from English.

Key Results and Insights

Sentence Encoders as Word Encoders

A significant finding is that LASER, despite being trained for sentence-level representations, contains internal hidden states that are remarkably effective for word-level tasks. In zero-shot NER for German, LASER-top achieved a state-of-the-art F1 score of 61.7, outperforming Transformer-based models.

Architecture Performance Trade-offs

While LASER-top performed exceptionally well in specific transfers, mBERT remained the most robust overall winner across 6 out of 9 tasks. For highly distant languages like Basque, BERT achieved a 0.403 F1, significantly higher than XLM (0.08) or LASER-top (0.24). This suggests that the massive scale of BERT's pretraining provides a robustness advantage for linguistically diverse targets.

Knowledge Sharing and Contextualization

In joint training scenarios (training on multiple languages simultaneously), LASER-based models showed a higher performance "boost" than BERT. This indicates that translation-based pretraining may create a more truly language-agnostic space than masked language modeling. Furthermore, for low-resource languages, any form of contextualization (BERT or LASER) provided a massive performance jump over static embeddings like MUSE.

Conclusion

The study demonstrates that BiLSTM-based translation pretraining (LASER) is a powerful alternative to Transformer-based modeling for multilingual word embeddings. While BERT remains a robust general-purpose baseline, LASER's ability to align languages through translation offers superior knowledge sharing in joint training. Future work may involve hybrid approaches that combine the strengths of masked language modeling with the cross-lingual alignment of translation-based objectives.

Summary updated

Abstract

The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-of-the-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multiple baselines and show that it performs at or above state-of-the-art level in zero-shot transfer settings. Finally, we show that our method allows for better knowledge sharing across languages in a joint training setting.
BibTeX
@inproceedings{heijden2020comparison,
  author = {Niels van der Heijden and Samira Abnar and Ekaterina Shutova},
  title = {{A Comparison of Architectures and Pretraining Methods for Contextualized Multilingual Word Embeddings}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=DonQu5SEiH},
}
Figure