Multilingual Pre-training with Universal Dependency Learning

Kailai Sun, Zuchao Li, hai zhaoNeurIPS 2021 · OpenReview

TL;DR

Current multilingual language models often ignore the underlying structural "skeleton" of language, treating text as a flat sequence of tokens. This paper introduces a novel pre-training approach that explicitly teaches models Universal Dependency (UD) parsing, integrating these syntactic structures directly into the model's internal representations. The result is a family of models (UD-BERT and UD-XLM-R) that outperform standard baselines on cross-lingual understanding tasks and can function as high-accuracy, standalone parsers for dozens of languages.

What Problem Does This Paper Address?

Pre-trained Language Models (PrLMs) like BERT and XLM-R have revolutionized NLP, but their multilingual versions often struggle with "low-resource" languages where training data is scarce. While these models are excellent at capturing statistical patterns in plain text, they often overlook explicit linguistic structures—like how a verb relates to its subject—that are remarkably consistent across different languages.

Word order varies wildly between languages; for instance, English follows a Subject-Verb-Object pattern, while Japanese uses Subject-Object-Verb. However, the functional relationship between words (dependency) is far more universal. The authors argue that by forcing a model to learn these universal syntactic "clues" during pre-training, it can develop a more robust, language-agnostic understanding of text.

Methods: Integrating Syntax into the Model

The authors propose an architecture that moves beyond simple multi-task learning. Instead of just having the model predict syntax as a secondary objective, they make syntax an integral part of the model's internal representation.

  1. Universal Structure Learning (USL) Layer: The researchers add a deep biaffine dependency parser on top of a standard Transformer stack. This layer learns to predict the "arc" (which words are connected) and the "label" (the type of relationship, like nsubj or obj) between tokens.
  2. Universal Structure Integration (USI) Layer: This is the core innovation. The model doesn't just output a parse tree; it uses the predicted tree to refine its own hidden states. It treats the parser's score matrices as a form of "syntactic attention," weighting the representations of related words more heavily.
  3. Subword-to-Word Mapping: Since UD trees are defined at the word level but BERT-style models use subword tokens, the authors developed a conversion strategy to ensure the syntax remains consistent across different tokenization schemes.
The model architecture for UD-PrLM, showing how the USL and USI layers interact with the standard Transformer stack.

Datasets

  • Pre-training: The models were trained on Wikipedia (104 languages) and CommonCrawl data.
  • Syntactic Supervision: They utilized the Universal Dependencies (UD) Treebanks (v2.2 and v2.7), which provide annotated syntactic structures for over 60 languages.
  • Evaluation: The models were tested on XNLI (Cross-lingual Natural Language Inference), XQuAD (Cross-lingual Question Answering), and various parsing benchmarks including SPMRL (morphologically-rich languages) and the Penn Treebank.

Key Insights

  • Syntax as a Representation: Most previous work treats syntax as a target task. This paper treats it as a feature. By integrating the parse tree back into the hidden states, the model's final output is inherently "syntax-aware."
  • Interpretability: Because the model contains an explicit parsing layer, researchers can inspect the internal "dependency tree" the model is using to solve a task, reducing the "black box" nature of standard Transformers.
  • Universal Signal: Explicitly training on UD acts as a powerful cross-lingual signal. It helps the model map concepts between languages not just by vocabulary, but by how those languages function structurally.

Key Results

The "UD-enhanced" models consistently outperformed their vanilla counterparts across the board:

  • NLU Benchmarks: UD-BERT showed an average improvement of 1.1% over mBERT on the XNLI dataset. UD-XLM-R similarly outperformed the standard XLM-R.
  • Question Answering: On XQuAD, the improvements were even more pronounced in specific languages; for example, UD-XLM-R saw a massive 6.6 F1 score boost in Arabic compared to the baseline.
  • Parsing Performance: Even without additional fine-tuning, UD-BERT achieved parsing accuracy (UAS/LAS) comparable to or better than dedicated state-of-the-art parsers.
Table 1: Results on cross-lingual text classification (XNLI), showing UD-variants consistently outperforming baselines.

Limitations and Future Work

While the results are impressive, the authors note a few areas for growth:

  • Supervision Dependency: The method relies on the availability of UD Treebanks. While UD covers many languages, it is not exhaustive.
  • Computational Overhead: Adding the USL and USI layers increases the complexity of the model, though the authors argue the performance gains justify the cost.
  • Future Directions: The authors suggest exploring how this structural integration might benefit generative tasks (like translation) or even more complex reasoning tasks where logical structure is paramount.

Summary updated

Abstract

The pre-trained language model (PrLM) demonstrates domination in downstream natural language processing tasks, in which multilingual PrLM takes advantage of language universality to alleviate the issue of limited resources for low-resource languages. Despite its successes, the performance of multilingual PrLM is still unsatisfactory, when multilingual PrLMs only focus on plain text and ignore obvious universal linguistic structure clues. Existing PrLMs have shown that monolingual linguistic structure knowledge may bring about better performance. Thus we propose a novel multilingual PrLM that supports both explicit universal dependency parsing and implicit language modeling. Syntax in terms of universal dependency parse serves as not only pre-training objective but also learned representation in our model, which brings unprecedented PrLM interpretability and convenience in downstream task use. Our model outperforms two popular multilingual PrLM, multilingual-BERT and XLM-R, on cross-lingual natural language understanding (NLU) benchmarks and linguistic structure parsing datasets, demonstrating the effectiveness and stronger cross-lingual modeling capabilities of our approach.
BibTeX
@inproceedings{sun2021multilingual,
  author = {Kailai Sun and Zuchao Li and hai zhao},
  title = {{Multilingual Pre-training with Universal Dependency Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=5JIAKpVrmZK},
}
Figure