TL;DR
This paper investigates how to improve the performance of multilingual language models (like BERT) on low-resource languages. They find that adapting the model's vocabulary and transliterating text into Latin script can significantly boost performance, especially for languages with non-Latin scripts, but these two methods can sometimes interfere with each other.
What Problem Does This Paper Address?
Natural Language Processing (NLP) research has largely focused on high-resource languages like English, where abundant data is available. However, the vast majority of the world's languages are low-resource, meaning they have very little labeled data for training NLP models. Multilingual language models (MLMs) offer a promising solution by leveraging data from many languages to improve performance on low-resource ones. However, standard MLMs often struggle with low-resource languages due to issues like poor tokenization (how text is broken down into units the model understands) and a bias towards Latin-script languages in their training data and internal vocabulary. This paper aims to empirically study and compare two specific adaptation techniques – vocabulary augmentation and script transliteration – to address these limitations.
Methods
The paper focuses on two primary adaptation techniques for specializing multilingual models:
- Vocabulary Augmentation (VA): This method, building on prior work by Chau et al. (2020), addresses poor tokenization by augmenting the MLM's existing vocabulary. It involves training a new wordpiece vocabulary on unlabeled data from the target language and then adding the most frequent new wordpieces (that would otherwise be "unknown" tokens) to the original vocabulary. The model is then further pre-trained on this augmented vocabulary.
- Script Transliteration: Inspired by Muller et al. (2021), this technique involves converting text from non-Latin scripts into the Latin script. The hypothesis is that this makes the text more compatible with the inherently Latin-centric vocabulary and architecture of many MLMs, thereby facilitating cross-lingual transfer.
The researchers also explore a "mix-in" paradigm, where these adaptations are applied sequentially, and investigate their interaction.
Datasets
The study uses nine typologically diverse low-resource languages, chosen for their varied scripts and representation levels in existing MLMs. These languages include:
- Labeled Datasets:
- Universal Dependencies (UD) v2.5 for Part-of-Speech (POS) tagging and dependency parsing.
- WikiAnn dataset for Named Entity Recognition (NER).
- Unlabeled Datasets: Articles from Wikipedia, downsampled to simulate low-resource settings, used for additional pre-training.
The paper explicitly notes that they do not introduce new benchmarks but rather evaluate existing methods on a diverse set of languages and tasks.
Key Insights and Ideas
- Non-Latin Scripts are a Major Hurdle: The paper strongly reinforces that MLMs' inherent bias towards Latin scripts is a significant bottleneck for performance on languages using other writing systems.
- Vocabulary Augmentation is Effective, Especially for Non-Latin Scripts: VA proves to be a robust method for improving tokenization and, consequently, downstream task performance, particularly for languages with non-Latin scripts and those less represented in the original MLM pre-training.
- Transliteration Offers a Complementary Approach: Script transliteration also shows promise, especially for non-Latin scripts, by aligning the input data with the model's Latin-centric nature.
- Adaptation is Best Done Before Final Pre-training: The "mix-in" stage (between initial and final pre-training) is identified as a crucial point for applying specialization techniques like VA and transliteration to achieve significant gains.
- VA and Transliteration Can Conflict: A key finding is that combining VA and transliteration doesn't always yield additive benefits and can sometimes lead to degraded performance, suggesting a potential conflict or redundancy between the two methods.
Key Results
- Vocabulary Augmentation (VA):
- Achieved strong results across all tested languages and tasks, often outperforming unadapted MBERT and language-adaptive pretraining (LAPT).
- Showed particularly significant improvements for languages with non-Latin scripts (e.g., Uyghur, Urdu), with up to a 57% relative error reduction in NER for Uyghur.
- The gains from VA were strongly correlated with improvements in vocabulary coverage (reduction in "unknown" tokens).
- Script Transliteration:
- Also provided strong improvements over LAPT for non-Latin low-resource languages.
- For Meadow Mari and Uyghur (Type 2 languages), transliteration combined with VA showed mixed results, sometimes underperforming VA alone or LAPT with transliteration.
- Interaction:
- VA and transliteration interacted negatively in several experiments. For instance, in NER tasks for Uyghur, VA with transliteration lagged behind both VA alone and LAPT with transliteration. This suggests that transliteration might implicitly provide some of the benefits of vocabulary augmentation, or that the two methods might conflict in how they modify the model's representation.
Limitations and Future Work
- Interaction Complexity: The negative interaction between VA and transliteration is noted as a key area for future research. The authors hypothesize this could be due to transliteration implicitly augmenting vocabulary or conflicting character representations.
- Broader Model Applicability: The study primarily uses MBERT as a case study. Future work could extend these findings to other multilingual models like mT5 and XLM-R to see if similar patterns hold.
- Unlabeled Data Size: While the study uses small unlabeled datasets to mimic low-resource scenarios, further investigation into the impact of different unlabeled data sizes is warranted.
- Optimal Mix-in Configurations: Exploring the optimal amount of vocabulary augmentation when combined with transliteration could yield more synergistic results.
- Linguistically-Informed Transliteration: Designing more sophisticated, linguistically-informed transliteration schemes could potentially improve their compatibility with VA.
Summary updated