Abstract
Multilingual Pretrained Language Models (mPLMs) like mBERT are limited to a small fraction of the world's languages. For the thousands of "unseen" low-resource languages (LRLs), adaptation typically involves either Vocabulary Augmentation (VA) or Transliteration (TL). While VA preserves lexical identity, TL facilitates cross-lingual transfer; however, both have distinct drawbacks. We present ScriptMix, a framework that combines these strategies using a dual-script corpus and a specialized adapter architecture. By separating language modules for original and transliterated scripts and introducing an improved fusion mechanism (AdapterFusion+), ScriptMix achieves superior performance in part-of-speech tagging and dependency parsing for LRLs.
Introduction: The Challenge of Unseen Languages
Despite the power of mPLMs, they cover only about 100 of the world's 6,500+ languages. Adapting these models to unseen languages usually relies on two competing strategies. Vocabulary Augmentation (VA) adds original script tokens to the model, preserving word identity but failing to leverage knowledge from high-resource languages (HRLs) due to lack of overlap. Conversely, Transliteration (TL) converts LRL scripts into HRL scripts (e.g., Latin), enabling semantic transfer but introducing "false positives" where distinct LRL words map to the same HRL token.
ScriptMix is motivated by the hypothesis that these two methods are complementary: TL excels at semantic transfer, while VA is superior at disambiguating words that appear identical after transliteration.
The ScriptMix Framework
The core challenge in combining VA and TL is "gradient conflict," where simultaneous training on different scripts in a single module leads to interference. ScriptMix addresses this through three primary innovations:
- Dual-Script Corpus: The model is trained on a mixture of the original script and its transliterated counterpart.
- Language Module Separation: Instead of a shared adapter, ScriptMix employs separate adapters for the original script ($LA_s$) and the transliterated script ($LA_t$). This architectural separation allows the model to learn script-specific features without interference.
- AdapterFusion+: To integrate the outputs of $LA_s$ and $LA_t$, we introduce an MLP-based fusion layer. This replaces standard attention-based fusion, which we found to have a harmful inductive bias toward inputs that most closely resemble the adapter's output.
Experimental Evaluation
We evaluated ScriptMix on five diverse low-resource languages: Uyghur, Old Church Slavonic, Russia Buriat, Erzya, and Amharic. Tasks included Part-of-Speech (POS) tagging and Dependency Parsing (DEP) using Universal Dependencies treebanks.
Key Results
ScriptMix consistently outperformed the standard MAD-X baseline across all settings:
- POS Tagging: Accuracy gains reached up to 14%, particularly in Amharic and Old Church Slavonic.
- Dependency Parsing: Labeled Attachment Scores (LAS) improved by up to 5.6%.
- Few-Shot Robustness: The framework provided significant performance lifts even with as few as 16 labeled training sentences.
Analysis of the fusion weights confirmed that the model dynamically utilizes both scripts, often relying on the original script in specific layers to resolve ambiguities introduced by transliteration.
Conclusion
ScriptMix demonstrates that combining transliteration and vocabulary augmentation is not only possible but highly effective when managed through appropriate architectural separation. By mitigating gradient conflict and refining the fusion process, ScriptMix provides a robust path forward for extending the reach of multilingual models to the world's many underserved languages.
Summary updated