75 Languages, 1 Model: Parsing Universal Dependencies Universally

Daniel Kondratyuk, Milan StrakaEMNLP 2019 · DOI

Abstract

This paper introduces UDify, a single neural network capable of parsing 75 different languages into Universal Dependencies (UD). By leveraging a single multilingual BERT (mBERT) model fine-tuned on 124 concatenated datasets, the authors demonstrate that a unified multilingual approach is not only feasible but often superior to specialized, language-specific models. UDify achieves state-of-the-art results across various tasks, including part-of-speech tagging, morphological analysis, lemmatization, and dependency parsing, with particularly significant gains for low-resource languages through cross-lingual transfer.

1. Introduction

Traditional natural language processing (NLP) pipelines often require separate models for each language. This approach suffers from two primary drawbacks: data scarcity for low-resource languages and the operational inefficiency of maintaining hundreds of distinct models. UDify addresses these issues by utilizing the shared underlying structures of human languages within a single, massively multilingual model. By training on the Universal Dependencies (UD) v2.3 corpus—spanning 75 languages and 124 treebanks—UDify proves that "one model to rule them all" can effectively capture universal linguistic patterns.

2. Methodology

The core of UDify is built upon multilingual BERT (mBERT), a Transformer-based model pre-trained on 104 Wikipedias. The researchers extended this architecture to support multi-task learning and universal cross-lingual transfer.

2.1 Multi-Task Architecture

UDify simultaneously predicts Universal Part-of-Speech (UPOS) tags, morphological features (UFeats), lemmas, and dependency trees. To handle lemmatization efficiently, the model predicts "edit scripts"—sequences of operations to transform an input word into its base form—rather than generating characters individually.

The UDify architecture, showing the integration of multilingual BERT with task-specific heads for universal dependency parsing.

2.2 Layer Attention and Training

Instead of relying solely on the final layer of BERT, UDify employs Layer Attention, learning a task-specific weighted average of all 12 layers. This allows the model to extract low-level morphological features from earlier layers and high-level syntactic information from later ones. Training involved concatenating all 124 UD treebanks into a single dataset, supported by heavy regularization techniques like word-level dropout and discriminative fine-tuning to prevent overfitting to high-resource languages.

3. Results and Analysis

UDify met or exceeded state-of-the-art (SOTA) scores for Unlabeled Attachment Score (UAS) and Labeled Attachment Score (LAS) across the majority of the 75 languages tested.

3.1 Cross-Lingual Transfer

The most dramatic improvements were observed in low-resource languages. For instance, Kazakh saw its parsing accuracy (LAS) jump from approximately 33% to 63% when trained alongside related high-resource languages. This "Slavic synergy" and similar cross-lingual effects highlight the model's ability to transfer structural knowledge across the language family tree.

3.2 Probing Layer Preferences

Analysis of the learned layer weights revealed that different linguistic tasks utilize different depths of the BERT network. While morphology (UFeats) shows relative importance in the very first and last layers, complex syntax (Dependency Parsing) relies more heavily on the middle-to-late layers as the representation becomes more abstract.

Learned layer attention weights across the 12 BERT layers for different linguistic tasks, illustrating how different tasks "live" in different parts of the network.

4. Discussion and Conclusion

UDify demonstrates that a simple, unified architecture can outperform complex, language-specific systems. By reducing the number of required models from dozens to one, it offers a highly efficient path for universal NLP. While the model shows slight trade-offs in lemmatization due to the omission of character-level embeddings for speed, its zero-shot capabilities and massive performance boosts for low-resource languages mark a significant step forward in universal dependency parsing. Future work may explore injecting this syntactic bias into other downstream tasks to further improve multilingual language understanding.

Summary updated

BibTeX
@inproceedings{kondratyuk2019languages,
  author = {Daniel Kondratyuk and Milan Straka},
  title = {{75 Languages, 1 Model: Parsing Universal Dependencies Universally}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2019},
  doi = {10.18653/v1/d19-1279},
}
Figure