Improving the Performance of UDify with Linguistic Typology Knowledge

Chinmay ChoudharyNAACL 2021

TL;DR

This paper enhances UDify, a state-of-the-art language-agnostic dependency parser, by incorporating linguistic typology knowledge. By adding an auxiliary task to predict typology features, they significantly improve the model's cross-lingual transfer capabilities, especially for low-resource languages.

What Problem Does This Paper Address?

Dependency parsing, a crucial task in Natural Language Processing (NLP) for understanding sentence structure, typically requires large amounts of labeled data. This data scarcity poses a significant challenge for low-resource languages, where such datasets are rare. While multilingual models like UDify can generalize to these languages, their performance still lags behind high-resource languages. This paper aims to bridge this performance gap by leveraging linguistic typology, which describes systematic cross-linguistic variation in language structure.

Methods

The authors build upon UDify, a multilingual BERT-based dependency parser that simultaneously handles four tasks: UPOS tagging, UFeats tagging, Lemmatization, and Dependency Parsing. Their core innovation is the addition of a fifth auxiliary task: Linguistic Typology Prediction. This is implemented as a simple feed-forward neural network that takes the mBERT encoder's output and predicts binary typology features from the URIEL database. This typology predictor is trained jointly with the other tasks in a multi-task learning framework. The key advantage of this approach, as highlighted by the authors, is that it induces typology knowledge into the model rather than directly feeding it as input, making it applicable even when typology features are missing for a language.

Datasets

The primary dataset used for training is a large, joint polyglot corpus created by concatenating all training treebanks available in UDv2.5. For evaluation, the authors use all 80 test treebanks from UDv2.3. The URIEL database is used as the source of linguistic typology features.

Key Insights and Ideas

  • Typology as a "Hint" for Syntax: The paper hypothesizes that linguistic typology, particularly syntactic typology, can act as a valuable "hint" for dependency parsing, which is inherently a syntactic task.
  • Multi-task Learning for Knowledge Induction: Instead of directly inputting typology features, the authors use multi-task learning to "induce" this knowledge. This allows the model to learn how to leverage typology without requiring complete typology data for every language.
  • Syntactic Typology is Key: The experiments suggest that focusing on syntactic typology features is more beneficial for dependency parsing than including semantic or phonological features, likely because dependency parsing itself is a syntactic task.
  • Few-Shot vs. Zero-Shot Performance: The proposed method shows significant improvements in few-shot learning scenarios but does not lead to improvements in zero-shot scenarios, indicating that some data is still beneficial for the model to fully leverage the typology knowledge.

Key Results

The proposed UDify+Typology-predictor model, specifically the UDify-w-Syntax variant, demonstrates improved performance across various metrics (LAS, UAS) compared to the baseline UDify and other variants.

  • Overall Improvement: Table 1 shows that UDify-w-Syntax achieves higher average LAS scores across all 80 test treebanks compared to the baseline UDify.
  • Low-Resource Language Gains: The paper highlights that the improvements are particularly pronounced for low-resource languages, where the gap between high-resource and low-resource performance is reduced. Figure 2 illustrates this, showing UDify+Syntax having a more uniform performance across treebanks of varying sizes.
  • Syntactic Focus: The UDify-w-Syntax variant consistently outperforms UDify-w-Syntactic+Semantic and UDify-w-All on most treebanks, suggesting the relevance of syntactic typology features.
  • Statistical Significance: Table 3 shows statistically significant correlations between the auxiliary typology F1 scores and the improvements in UAS for UDify+Syntax over UDify.

Limitations and Future Work

  • Zero-Shot Performance: The paper notes that the auxiliary task does not lead to improvements in zero-shot learning scenarios. This suggests that while typology knowledge helps in transferring from languages with some data, it doesn't fully compensate for the complete absence of data.
  • Feature Relevance: The authors found that syntactic typology features were most relevant. Future work could explore more nuanced ways to select or weight different types of typology features based on the target task.
  • Broader Typology Databases: The study relies on the URIEL database. Future work could investigate the impact of using other or more comprehensive typology databases.
  • Beyond Dependency Parsing: The authors suggest that this approach could be extended to other NLP tasks where cross-lingual transfer is important and where linguistic typology might be relevant.

Summary updated

BibTeX
@inproceedings{choudhary2021improving,
  author = {Chinmay Choudhary},
  title = {{Improving the Performance of UDify with Linguistic Typology Knowledge}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure