Multilingual Dependency Parsing for Low-Resource African Languages: Case Studies on Bambara, Wolof, and Yoruba

Cheikh M. Bamba DioneACL 2021

TL;DR

This paper tackles the challenge of building dependency parsers for low-resource African languages by leveraging knowledge transfer from high-resource languages using a multilingual BERT model. They demonstrate that combining treebanks from languages with historical contact can significantly improve parsing accuracy for languages with very limited annotated data.

What Problem Does This Paper Address?

Natural Language Processing (NLP) applications rely heavily on annotated data, particularly treebanks, for tasks like syntactic parsing. However, creating these treebanks is a time-consuming and expensive process, leading to a severe lack of annotated data for most of the world's languages, especially those in Africa. This "low-resource" problem hinders the development of NLP tools for these languages. This paper aims to bridge this gap by developing a method to transfer syntactic knowledge from well-resourced languages (like English and French) to extremely low-resource African languages (Bambara, Wolof, and Yoruba) using multilingual models.

Methods

The core of the methodology is a multilingual multi-task model built upon the UDify framework. UDify utilizes a pre-trained multilingual BERT model to generate contextual embeddings for input sentences. This is followed by a graph-based dependency parser employing biaffine attention, a state-of-the-art technique for dependency parsing. The key innovation lies in the cross-lingual transfer learning approach:

  • Multilingual BERT: This model is pre-trained on a massive amount of text from many languages, allowing it to learn general linguistic patterns.
  • Multi-task Learning: The model is trained to predict multiple Universal Dependencies (UD) annotations simultaneously: part-of-speech (UPOS), morphological features (FEATS), lemmas (LEM), and dependency trees (DEPS).
  • Treebank Combination: The researchers combine existing UD treebanks from high-resource languages (English, French, Norwegian) with the limited treebanks available for the low-resource African languages. This combined dataset is used to train a single, unified parsing model.
  • Language Contact Hypothesis: A central assumption is that languages with historical or contemporary contact might share structural similarities, making them good candidates for knowledge transfer.

Datasets

The study utilizes Universal Dependencies (UD) v2.3 treebanks.

  • High-resource languages: English (multiple treebanks: EWT, GUM, ParTUT), French (multiple treebanks: FTB, ParTUT, Sequoia, Spoken), and Norwegian (Bokmål).
  • Low-resource languages: Bambara (CRB), Wolof (WTB), and Yoruba (YTB).

It's important to note that for Bambara and Yoruba, only test set data was available, making them ideal for zero-shot learning experiments. Wolof had a small training, development, and test set.

Key Insights and Ideas

  • Contact is Key (Sometimes): The paper strongly suggests that incorporating treebanks from languages with historical or contemporary contact with the target low-resource language can lead to better parsing performance than using unrelated languages. This highlights the importance of linguistic proximity in transfer learning.
  • Multilingual Models Outperform Monolingual (for Low-Resource): For the low-resource languages, the multilingual models consistently showed improved parsing quality compared to a monolingual model trained solely on the limited available data. This validates the effectiveness of cross-lingual transfer.
  • Norwegian as a Control: Using Norwegian, a language with no direct contact or genetic relation to the target African languages, served as a crucial control to isolate the effect of language contact. Removing it from the training data had minimal negative impact, further supporting the contact hypothesis.
  • Yoruba Anomaly: The experiment where Wolof data was excluded from training surprisingly led to an improvement in Yoruba's parsing quality. This challenges the initial assumption about genetic relatedness and suggests that the classification of genetic relationships might not always reflect actual linguistic similarities or that data limitations play a significant role.

Key Results

  • Significant Improvements: The multilingual models achieved notable improvements in parsing accuracy (UAS and LAS) for Bambara (around 5% and 4.38% increase, respectively) and Yoruba (around 11.37% and 12.34% increase, respectively) compared to baseline results (likely monolingual or less optimal combinations).
  • Wolof Gains: For Wolof, the multilingual approach resulted in a 2.23% increase in UAS and 3.19% in LAS compared to a monolingual model, demonstrating the benefit of transfer learning.
  • Impact of English Exclusion: Excluding English treebanks from training caused a slight drop for Bambara but a substantial decrease for Yoruba, underscoring English's contribution.
  • Impact of French Exclusion: Excluding French treebanks led to a decrease in parsing quality for Bambara and Wolof, but had no significant impact on Yoruba.
  • Wolof Data Crucial for Wolof: Removing Wolof's own training data resulted in a drastic drop in parsing accuracy for Wolof itself, as expected.

Limitations and Future Work

  • Yoruba/Wolof Genetic Relationship: The paper acknowledges that the observed results for Yoruba when Wolof data was excluded might challenge the assumed genetic relationship. They suggest this could be due to classification issues or insufficient data for Yoruba, warranting further investigation.
  • Data Scarcity: The fundamental limitation remains the extremely small size of the available UD treebanks for the low-resource languages, particularly the lack of training data for Bambara and Yoruba.
  • Future Work: The authors suggest further investigation into the Yoruba/Wolof anomaly and exploring other language combinations to better understand the impact of genetic and contact relationships. They also implicitly point towards the need for more annotated data for these languages.

Summary updated

BibTeX
@inproceedings{dione2021multilingual,
  author = {Cheikh M. Bamba Dione},
  title = {{Multilingual Dependency Parsing for Low-Resource African Languages: Case Studies on Bambara, Wolof, and Yoruba}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure