TL;DR
Current multilingual language models often ignore the underlying structural "skeleton" of language, treating text as a flat sequence of tokens. This paper introduces a novel pre-training approach that explicitly teaches models Universal Dependency (UD) parsing, integrating these syntactic structures directly into the model's internal representations. The result is a family of models (UD-BERT and UD-XLM-R) that outperform standard baselines on cross-lingual understanding tasks and can function as high-accuracy, standalone parsers for dozens of languages.
What Problem Does This Paper Address?
Pre-trained Language Models (PrLMs) like BERT and XLM-R have revolutionized NLP, but their multilingual versions often struggle with "low-resource" languages where training data is scarce. While these models are excellent at capturing statistical patterns in plain text, they often overlook explicit linguistic structures—like how a verb relates to its subject—that are remarkably consistent across different languages.
Word order varies wildly between languages; for instance, English follows a Subject-Verb-Object pattern, while Japanese uses Subject-Object-Verb. However, the functional relationship between words (dependency) is far more universal. The authors argue that by forcing a model to learn these universal syntactic "clues" during pre-training, it can develop a more robust, language-agnostic understanding of text.
Methods: Integrating Syntax into the Model
The authors propose an architecture that moves beyond simple multi-task learning. Instead of just having the model predict syntax as a secondary objective, they make syntax an integral part of the model's internal representation.
- Universal Structure Learning (USL) Layer: The researchers add a deep biaffine dependency parser on top of a standard Transformer stack. This layer learns to predict the "arc" (which words are connected) and the "label" (the type of relationship, like nsubj or obj) between tokens.
- Universal Structure Integration (USI) Layer: This is the core innovation. The model doesn't just output a parse tree; it uses the predicted tree to refine its own hidden states. It treats the parser's score matrices as a form of "syntactic attention," weighting the representations of related words more heavily.
- Subword-to-Word Mapping: Since UD trees are defined at the word level but BERT-style models use subword tokens, the authors developed a conversion strategy to ensure the syntax remains consistent across different tokenization schemes.
Datasets
- Pre-training: The models were trained on Wikipedia (104 languages) and CommonCrawl data.
- Syntactic Supervision: They utilized the Universal Dependencies (UD) Treebanks (v2.2 and v2.7), which provide annotated syntactic structures for over 60 languages.
- Evaluation: The models were tested on XNLI (Cross-lingual Natural Language Inference), XQuAD (Cross-lingual Question Answering), and various parsing benchmarks including SPMRL (morphologically-rich languages) and the Penn Treebank.
Key Insights
- Syntax as a Representation: Most previous work treats syntax as a target task. This paper treats it as a feature. By integrating the parse tree back into the hidden states, the model's final output is inherently "syntax-aware."
- Interpretability: Because the model contains an explicit parsing layer, researchers can inspect the internal "dependency tree" the model is using to solve a task, reducing the "black box" nature of standard Transformers.
- Universal Signal: Explicitly training on UD acts as a powerful cross-lingual signal. It helps the model map concepts between languages not just by vocabulary, but by how those languages function structurally.
Key Results
The "UD-enhanced" models consistently outperformed their vanilla counterparts across the board:
- NLU Benchmarks: UD-BERT showed an average improvement of 1.1% over mBERT on the XNLI dataset. UD-XLM-R similarly outperformed the standard XLM-R.
- Question Answering: On XQuAD, the improvements were even more pronounced in specific languages; for example, UD-XLM-R saw a massive 6.6 F1 score boost in Arabic compared to the baseline.
- Parsing Performance: Even without additional fine-tuning, UD-BERT achieved parsing accuracy (UAS/LAS) comparable to or better than dedicated state-of-the-art parsers.
Limitations and Future Work
While the results are impressive, the authors note a few areas for growth:
- Supervision Dependency: The method relies on the availability of UD Treebanks. While UD covers many languages, it is not exhaustive.
- Computational Overhead: Adding the USL and USI layers increases the complexity of the model, though the authors argue the performance gains justify the cost.
- Future Directions: The authors suggest exploring how this structural integration might benefit generative tasks (like translation) or even more complex reasoning tasks where logical structure is paramount.
Summary updated