Abstract
Ethan C. Chau, Lucy H. Lin, Noah A. Smith. Findings of the Association for Computational Linguistics: EMNLP 2020.
2020.
This paper demonstrates that "off-the-shelf" multilingual BERT (mBERT) can be significantly improved for low-resource language varieties through two simple adaptation strategies: additional language-specific pretraining (LAPT) and vocabulary augmentation (VA). By leveraging small amounts of unlabeled data and repurposing unused vocabulary slots, the authors achieved substantial gains in dependency parsing accuracy, particularly for languages that were either underrepresented or entirely absent from mBERT’s original training.
Introduction
Large-scale multilingual models like mBERT are designed to share knowledge across languages, theoretically helping low-resource languages benefit from high-resource ones. However, they face a transfer-dilution tradeoff: as more languages are added, the model’s capacity for each individual language decreases.
The authors identify a major gap in how these models handle two specific "underdog" categories:
- Type 1 (Low-resource): Languages that were in the original training set but had very little data (e.g., Irish).
- Type 2 (Unseen): Languages that were completely excluded from the original training (e.g., Maltese).
Because mBERT uses a fixed vocabulary based on its original training data, these languages often suffer from poor tokenization or "unknown" (UNK) tokens, making internal representations less effective for downstream tasks like dependency parsing.
Datasets and Language Types
The study focused on four typologically diverse language varieties, categorized by their relationship to the original mBERT training set.
- Irish (GA): A Type 1 language (present in mBERT but low-resource).
- Maltese (MT): A Type 2 language (entirely unseen by mBERT).
- Singlish (SING): An English-based creole. While its components are in mBERT, the variety itself is unique.
- Vietnamese (VI): Used as a "Type 0" control (well-represented in mBERT), with a simulated low-resource setting using only 5% of available Wikipedia data.
Methods
The researchers tested three main adaptation techniques:
- Language-Adaptive Pretraining (LAPT): Continuing mBERT's pretraining on a small, monolingual corpus of the target language.
- Vocabulary Augmentation (VA): mBERT’s vocabulary includes 99 "unused" slots. The authors assigned the most frequent wordpieces in the target language missing from mBERT to these slots, allowing the model to learn specific embeddings for important local terms.
- Tiered Vocabulary Augmentation (TVA): A variant of VA where the newly added vocabulary embeddings are trained with a higher learning rate to help them "catch up" quickly.
They evaluated these methods on dependency parsing, using a graph-based parser on top of the mBERT representations.
Key Results
The metrics used were Labeled Attachment Score (LAS) for dependency parsing. The results show that adaptation provides a massive boost, particularly for the unseen Maltese variety.
- Maltese (Type 2): Saw the most dramatic improvement. Base mBERT scored ~67.06, while the VA + FT approach jumped to 83.53—a nearly 40% relative error reduction.
- Irish (Type 1): Improved from ~68.19 (base) to 76.17 (VA + FT).
- Singlish & Vietnamese: Both saw significant gains (around 5-11% error reduction), confirming that even "known" languages benefit from local adaptation.
Discussion and Insights
The effectiveness of these methods is highly correlated with the language's relationship to the original model. The "unseen" Maltese saw the most dramatic improvements, while the "well-represented" Vietnamese saw the least.
Two major takeaways emerged: first, vocabulary is a bottleneck. Repurposing just 99 unused slots provided a measurable boost by reducing the reliance on fragmented subwords. Second, LAPT helps re-align knowledge. Additional pretraining on even a small corpus helps the model adjust its multilingual knowledge to the specific syntax and semantics of the target variety. Finally, while adapting representations is effective, further finetuning (FT) the entire model during the parsing task yields the best results, as it allows the model to specialize its layers for the specific grammar of the language.
Summary updated