Abstract
The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-of-the-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multiple baselines and show that it performs at or above state-of-the-art level in zero-shot transfer settings. Finally, we show that our method allows for better knowledge sharing across languages in a joint training setting.
Introduction
Most state-of-the-art NLP models require vast amounts of annotated data, which is unavailable for the majority of the world's languages. To bridge this "data gap," researchers utilize multilingual embeddings—mathematical representations of words in a shared vector space where semantically similar words across languages (e.g., "apple" and "manzana") are positioned closely together.
While static embeddings (one vector per word) were previously the standard, contextualized embeddings—where vectors change based on surrounding words—have significantly improved performance. However, much of this research has focused on single languages or sentence-level tasks. This paper investigates which architectures and pretraining methods are most effective for word-level tasks, such as Named Entity Recognition (NER) and Part-of-Speech (POS) tagging, particularly when transferring knowledge from high-resource languages like English to low-resource targets.
Methodology
The authors evaluate several major architectures, focusing on how different pretraining objectives influence cross-lingual transferability.
Transformer-based Models
The study examines Multilingual BERT (mBERT) and the Cross-lingual Language Model (XLM). These models utilize the Transformer architecture and are pretrained on massive Wikipedia datasets. mBERT primarily uses Masked Language Modeling (MLM), while XLM introduces Translation Language Modeling (TLM) to leverage parallel corpora for better alignment.
LASER-based Models (The Proposed Approach)
LASER (Language-Agnostic SEntence Representations) is a 5-layer BiLSTM encoder originally designed to translate sentences into a language-agnostic space. The authors propose extracting word-level information from this sentence encoder using two methods:
- LASER-top: Utilizing the hidden states from the final layer of the encoder to represent sub-words.
- LASER-elmo: Applying a weighted average of all five layers (inspired by the ELMo architecture) to capture both syntactic and semantic information.
Experimental Setup
The models were evaluated on the CoNLL-2002 and CoNLL-2003 shared tasks. The primary evaluation metric was zero-shot transfer: training a model on English data and testing it on target languages (Dutch, German, Spanish, Hungarian, and Basque) without any target-language training labels. Hungarian and Basque were specifically chosen to test model robustness on languages linguistically distant from English.
Key Results and Insights
Sentence Encoders as Word Encoders
A significant finding is that LASER, despite being trained for sentence-level representations, contains internal hidden states that are remarkably effective for word-level tasks. In zero-shot NER for German, LASER-top achieved a state-of-the-art F1 score of 61.7, outperforming Transformer-based models.
Architecture Performance Trade-offs
While LASER-top performed exceptionally well in specific transfers, mBERT remained the most robust overall winner across 6 out of 9 tasks. For highly distant languages like Basque, BERT achieved a 0.403 F1, significantly higher than XLM (0.08) or LASER-top (0.24). This suggests that the massive scale of BERT's pretraining provides a robustness advantage for linguistically diverse targets.
Knowledge Sharing and Contextualization
In joint training scenarios (training on multiple languages simultaneously), LASER-based models showed a higher performance "boost" than BERT. This indicates that translation-based pretraining may create a more truly language-agnostic space than masked language modeling. Furthermore, for low-resource languages, any form of contextualization (BERT or LASER) provided a massive performance jump over static embeddings like MUSE.
Conclusion
The study demonstrates that BiLSTM-based translation pretraining (LASER) is a powerful alternative to Transformer-based modeling for multilingual word embeddings. While BERT remains a robust general-purpose baseline, LASER's ability to align languages through translation offers superior knowledge sharing in joint training. Future work may involve hybrid approaches that combine the strengths of masked language modeling with the cross-lingual alignment of translation-based objectives.
Summary updated