Abstract
This paper introduces UDify, a single neural network capable of parsing 75 different languages into Universal Dependencies (UD). By leveraging a single multilingual BERT (mBERT) model fine-tuned on 124 concatenated datasets, the authors demonstrate that a unified multilingual approach is not only feasible but often superior to specialized, language-specific models. UDify achieves state-of-the-art results across various tasks, including part-of-speech tagging, morphological analysis, lemmatization, and dependency parsing, with particularly significant gains for low-resource languages through cross-lingual transfer.
1. Introduction
Traditional natural language processing (NLP) pipelines often require separate models for each language. This approach suffers from two primary drawbacks: data scarcity for low-resource languages and the operational inefficiency of maintaining hundreds of distinct models. UDify addresses these issues by utilizing the shared underlying structures of human languages within a single, massively multilingual model. By training on the Universal Dependencies (UD) v2.3 corpus—spanning 75 languages and 124 treebanks—UDify proves that "one model to rule them all" can effectively capture universal linguistic patterns.
2. Methodology
The core of UDify is built upon multilingual BERT (mBERT), a Transformer-based model pre-trained on 104 Wikipedias. The researchers extended this architecture to support multi-task learning and universal cross-lingual transfer.
2.1 Multi-Task Architecture
UDify simultaneously predicts Universal Part-of-Speech (UPOS) tags, morphological features (UFeats), lemmas, and dependency trees. To handle lemmatization efficiently, the model predicts "edit scripts"—sequences of operations to transform an input word into its base form—rather than generating characters individually.
2.2 Layer Attention and Training
Instead of relying solely on the final layer of BERT, UDify employs Layer Attention, learning a task-specific weighted average of all 12 layers. This allows the model to extract low-level morphological features from earlier layers and high-level syntactic information from later ones. Training involved concatenating all 124 UD treebanks into a single dataset, supported by heavy regularization techniques like word-level dropout and discriminative fine-tuning to prevent overfitting to high-resource languages.
3. Results and Analysis
UDify met or exceeded state-of-the-art (SOTA) scores for Unlabeled Attachment Score (UAS) and Labeled Attachment Score (LAS) across the majority of the 75 languages tested.
3.1 Cross-Lingual Transfer
The most dramatic improvements were observed in low-resource languages. For instance, Kazakh saw its parsing accuracy (LAS) jump from approximately 33% to 63% when trained alongside related high-resource languages. This "Slavic synergy" and similar cross-lingual effects highlight the model's ability to transfer structural knowledge across the language family tree.
3.2 Probing Layer Preferences
Analysis of the learned layer weights revealed that different linguistic tasks utilize different depths of the BERT network. While morphology (UFeats) shows relative importance in the very first and last layers, complex syntax (Dependency Parsing) relies more heavily on the middle-to-late layers as the representation becomes more abstract.
4. Discussion and Conclusion
UDify demonstrates that a simple, unified architecture can outperform complex, language-specific systems. By reducing the number of required models from dozens to one, it offers a highly efficient path for universal NLP. While the model shows slight trade-offs in lemmatization due to the omission of character-level embeddings for speed, its zero-shot capabilities and massive performance boosts for low-resource languages mark a significant step forward in universal dependency parsing. Future work may explore injecting this syntactic bias into other downstream tasks to further improve multilingual language understanding.
Summary updated