Abstract
The lack of annotated data in many languages is a well-known challenge within the field of multilingual natural language processing (NLP). Therefore, many recent studies focus on zero-shot transfer learning and joint training across languages to overcome data scarcity for low-resource languages. In this work we (i) perform a comprehensive comparison of state-of-the-art multilingual word and sentence encoders on the tasks of named entity recognition (NER) and part of speech (POS) tagging; and (ii) propose a new method for creating multilingual contextualized word embeddings, compare it to multiple baselines and show that it performs at or above state-of-the-art level in zero-shot transfer settings. Finally, we show that our method allows for better knowledge sharing across languages in a joint training setting.
Introduction
In Natural Language Processing (NLP), English often benefits from massive amounts of annotated data. However, for the thousands of other languages spoken globally, data is scarce. To bridge this gap, researchers utilize multilingual word embeddings, which project different languages into a shared mathematical space. While the field has transitioned from "static" embeddings (one vector per word) to "contextualized" embeddings (vectors that change based on surrounding context) in English, this transition is less understood in multilingual settings.
This study addresses two primary challenges:
- Comparing different massive multilingual architectures (LSTMs vs. Transformers) on word-level tasks like Named Entity Recognition (NER) and Part-of-Speech (POS) tagging.
- Investigating whether a model trained for sentence-level understanding (LASER) can be adapted to provide high-quality word-level representations.
Methods and Architectures
The researchers compared several established models and introduced new variations based on the LASER architecture.
The Baselines and Transformer Models
- Baselines: Standard Bag-of-Words (BPE BOW), a GRU-based encoder, and MUSE (static cross-lingual embeddings).
- Transformer Giants: Multilingual BERT and XLM (Cross-lingual Language Model). BERT utilizes Masked Language Modeling (MLM), while XLM introduces Translation Language Modeling (TLM) to leverage parallel data.
LASER-based Contenders
LASER is a 5-layer biLSTM trained on 93 languages using a translation objective. The authors propose two ways to extract word embeddings from it:
- LASER-top: Uses the hidden states of the final layer of the LASER encoder.
- LASER-elmo: Uses a weighted average of all five layers, inspired by the ELMo architecture.
To evaluate these, a Transformer-CRF architecture was used as the task-specific classification head. The table below summarizes the differences in pretraining data and tasks across the primary models.
Experimental Setup
Models were evaluated on NER and POS tagging using CoNLL 2002/2003 (English, Dutch, Spanish, German) and low-resource benchmarks (Hungarian, Basque). The primary training protocol involved training on English and testing on other languages (zero-shot) or joint training on a mix of languages.
Key Insights and Results
Contextualization and Zero-Shot Performance
Contextualized models (LASER-top, BERT) significantly outperformed static models like MUSE. Contextualization allows models to handle polysemy and casing more effectively by using surrounding sentence structure. For instance, LASER-top achieved a new state-of-the-art for English-to-German zero-shot NER, improving the F1-score by 2.8 points over previous methods.
Architecture Comparisons
- LASER-top vs. LASER-elmo: Surprisingly, LASER-top generally outperformed LASER-elmo. The authors suggest that the multi-layer weighted average in LASER-elmo led to overfitting on the source language (English), hindering cross-lingual generalization.
- BERT vs. LASER: BERT won in 6 out of 9 tasks, but LASER-top was a close second and outperformed BERT in several zero-shot POS tagging scenarios.
- XLM Performance: Despite its cross-lingual design, XLM did not consistently outperform standard multilingual BERT on these specific word-level tasks.
Joint Training and Knowledge Sharing
In joint training settings, LASER-based methods demonstrated superior "knowledge sharing." They showed more significant improvement than Transformer models when additional languages were added to the training set.
Conclusion
While LSTM-based LASER approaches are highly effective for zero-shot transfer and joint training, they still generally lag behind Transformer-based models like BERT in raw performance on most tasks. This suggests the Attention mechanism in Transformers provides a higher "ceiling" for capturing complex dependencies. Future work may involve combining the MLM objective of BERT with the translation-based pretraining of LASER to create models that better understand both internal language structure and cross-lingual semantic mappings.
Summary updated