TL;DR
This paper introduces "Rosita," a method for training a single language model on text from multiple languages to create multilingual contextual word representations. These "polyglot" representations significantly improve crosslingual transfer performance on tasks like dependency parsing, semantic role labeling, and named entity recognition, even for highly dissimilar languages.
What Problem Does This Paper Address?
The field of Natural Language Processing (NLP) has seen significant advancements with contextual word representations (CWRs), which capture word meaning based on its surrounding context, outperforming traditional static word embeddings. Simultaneously, crosslingual transfer aims to leverage knowledge from high-resource languages (like English) to improve performance on low-resource languages. However, a key question remained: can CWRs benefit from multilingual training, and how effectively can they bridge the gap between languages, especially dissimilar ones? Existing work showed mixed results, suggesting that simply combining data from different languages might not always be optimal. This paper addresses this by proposing and evaluating a method to create CWRs from a single, multilingual language model.
Methods
The core of this work is the "Rosita" method, which builds upon the ELMo architecture. Instead of training separate language models for each language, Rosita trains a single "polyglot" language model on text from multiple languages simultaneously. This allows for parameter sharing across languages during training.
The authors explore two main variants:
- MONOCHAR: A monolingual character-based language model.
- ROSITACHAR: A polyglot language model trained on a target language's data along with an equal amount of English data, using only character-level input. This is the primary focus for demonstrating the benefits of polyglot learning without relying on pre-aligned word embeddings.
- ROSITAWORD: A modified polyglot model that concatenates a pre-trained multilingual word embedding with the character CNN output before feeding it into the LSTM.
These models are then used to generate contextual word representations, which are fine-tuned for downstream tasks. The authors also compare these to traditional non-contextual multilingual word embeddings.
Datasets
The experiments utilize several datasets:
- English: Billion Word Benchmark for language model training.
- Traditional Chinese: Wiki and web data from the CONLL 2017 Shared Task for language model training.
- Simplified Chinese: Newswire text from Xinhua for language model training.
- Arabic: Newswire text from AFP for language model training.
- Universal Dependencies (UD) Treebanks: GUM (English), GSD (Chinese), and PADT (Arabic) for dependency parsing.
- Ontonotes: For Semantic Role Labeling (SRL) and Named Entity Recognition (NER) in English, Chinese, and Arabic.
The paper notes that for UD parsing, they use gold POS tags and word segmentations for evaluation, as well as predicted tags from other systems for direct comparison.
Key Insights and Ideas
- Polyglot CWRs offer a significant advantage: Even when traditional multilingual word embeddings hurt performance, polyglot CWRs can improve target language performance, suggesting they effectively tie dissimilar languages.
- Character-based polyglot models are powerful: The ROSITACHAR model, which doesn't rely on pre-aligned word embeddings, demonstrates strong crosslingual transfer capabilities, highlighting the power of learning shared representations from raw text.
- Crosslingual transfer is task-dependent: While polyglot training consistently helps, the degree of improvement varies across tasks (UD, SRL, NER) and language pairs.
- Unsupervised crosslingual connections: The success of ROSITACHAR suggests that language models can learn effective crosslingual connections from unlabeled data alone, without explicit bilingual dictionaries or parallel corpora.
Key Results
- Universal Dependency Parsing: Rosita models, particularly ROSITACHAR, achieved state-of-the-art results, outperforming previous systems, including those using monolingual CWRs and ensembles. For example, ROSITACHAR (CMN+ENG) achieved a LAS score of 77.63 for Chinese, significantly higher than the best prior work (62.83).
- Semantic Role Labeling (SRL): Multilingual word type embeddings performed better than monolingual ones in SRL. The ROSITAWORD model showed the highest performance on Chinese SRL. For Arabic SRL, while polyglot methods were slightly behind, they still showed strong results despite the smaller corpus size and language-specific tags.
- Named Entity Recognition (NER): Similar to other tasks, CWRs showed a strong advantage, with polyglot LMs providing a smaller additional benefit.
Limitations and Future Work
- Variability in benefit: The authors acknowledge that the benefit of polyglot training can be variable and task-dependent.
- Arabic SRL challenges: The poor results on Arabic SRL might be due to the smaller corpus size and a higher proportion of language-specific tags, which could limit the effectiveness of shared English-Arabic representations.
- Further study of polyglot models: The paper concludes by emphasizing that the success of polyglot language models warrants further study, particularly their use as unsupervised components in multilingual systems.
- Exploration of BERT: The authors mention that their initial exploration of multilingual BERT models raised preprocessing questions, deferring further investigation to future work.
Summary updated