TL;DR
This paper explores adapting large multilingual language models (PMMs) to a massive number of low-resource languages using only the New Testament as training data. Surprisingly, simple continued pretraining on this limited, domain-specific data yields the best results, outperforming more complex methods and demonstrating the Bible's utility for cross-lingual transfer.
What Problem Does This Paper Address?
Pretrained multilingual models (PMMs) like XLM-R are powerful tools for zero-shot cross-lingual transfer, meaning they can perform tasks in languages they weren't explicitly trained on. However, their performance is significantly better for languages well-represented in their pretraining data. For low-resource languages, especially those with limited or no digital text available, PMMs struggle. Existing adaptation methods to improve performance on these languages have typically been evaluated on datasets like Wikipedia or Common Crawl, which only cover a fraction of the world's languages. This leaves a critical gap: how can we adapt PMMs for the vast majority of languages that lack sufficient digital text, and what's the most effective way to do so? This paper tackles this by investigating the use of the New Testament, a text available in over 1600 languages, as a sole source of adaptation data.
Methods
The researchers evaluate several adaptation methods for PMMs, specifically focusing on XLM-R. The core idea is to adapt the model using only the New Testament text for each target language, without any labeled data in that language. The methods explored are:
- Continued Pretraining (+MLM, +TLM, +{M|T}LM): This involves further training the PMM on the New Testament data. +MLM uses masked language modeling, +TLM uses translation language modeling (leveraging parallel Bible verses), and +{M|T}LM combines both.
- Vocabulary Extension (+Extend): This method augments the PMM's subword vocabulary with new tokens learned from the target language's Bible text. This aims to better represent domain-specific words that might otherwise be fragmented.
- Adapters (+Adapters): This approach injects small, trainable adapter layers into the PMM. These adapters are trained on the target language's Bible text, allowing for parameter-efficient adaptation.
The study also includes a case study to disentangle the effects of domain (Bible vs. Wikipedia) and source language similarity (English vs. Russian) on adaptation performance.
Datasets
- Unlabeled Data: The primary source of unlabeled data is the Johns Hopkins University Bible Corpus (JHUBC), containing verse-aligned translations of the New Testament in over 1600 languages. For the experiments, the New Testament portion was used, averaging around 402k subword tokens per language for the 30 target languages.
- Labeled Data:
- For Named Entity Recognition (NER), data was sourced from the WikiAnn dataset.
- For Part-of-Speech (POS) tagging, data was taken from the Universal Dependencies Project.
- Case Study Data: For the case study, Wikipedia sentences were sampled to create a corpus of similar size to the Bible, and Russian was used as an alternative source language for labeled data.
Key Insights and Ideas
- The Bible as a Viable Resource: The most striking finding is that the New Testament, despite its small size and narrow domain, is a surprisingly effective resource for adapting PMMs to a large number of low-resource languages.
- Simplicity Wins: Contrary to expectations, the simplest adaptation method, continued pretraining (+MLM), consistently performed best across both NER and POS tagging tasks. This suggests that for extremely low-resource settings, focusing on robust language modeling with available text is more beneficial than complex architectural changes.
- Domain vs. Size: The case study revealed that while the Bible's domain is restrictive, it's still linguistically rich enough to boost performance, especially for tasks like POS tagging that are less sensitive to domain specificity than NER.
- Script Matters: Adaptation gains were generally higher for languages with Cyrillic script compared to Latin script, suggesting that greater linguistic distance from the pretraining data can sometimes lead to larger improvements when adaptation is successful.
- Low Baseline, High Gain: The paper observed that adaptation methods provide the largest improvements for languages with the weakest baseline performance. As baseline performance increases, the benefits of adaptation diminish.
Key Results
- NER: Continued pretraining methods (+MLM, +TLM, +{M|T}LM) achieved average gains of 3.93 to 6.29 F1 over the XLM-R baseline. Vocabulary extension and adapters performed worse than the baseline.
- POS Tagging: Continued pretraining methods achieved the highest increases, with average gains of 15.81 to 17.61 percentage points over the XLM-R baseline. Even the lowest performing adaptation method, +Adapters, showed a gain of 9.01 points.
- Case Study (Cyrillic NER):
- Using Russian as the labeled data source (instead of English) increased the baseline performance by 18.96 F1.
- Adapting with Wikipedia data (instead of the Bible) led to significant improvements, with +MLM showing an 11.85 F1 gain over the Russian baseline.
- The highest overall performance was achieved when using both Wikipedia data and Russian labeled data, with +MLM showing a 30.81 F1 gain over the English baseline.
Limitations and Future Work
- Task Selection: The study focused on NER and POS tagging, which are relatively low-level tasks. Future work could explore adaptation for higher-level tasks like question answering or natural language inference, which would require more diverse benchmarks.
- Bible Data Variability: The quality and quantity of Bible translations can vary significantly across languages, potentially affecting generalization.
- Model Bias: The use of PMMs means inheriting their inherent biases, which can be further amplified by using religious texts.
- Environmental Impact: The authors acknowledge the significant environmental cost of training large models.
- Future Directions: The paper suggests exploring more similar source languages for finetuning, leveraging linguistic knowledge for better adaptation, and investigating the impact of different PMM architectures. Extending benchmarks to truly low-resource languages is also highlighted as crucial.
Summary updated