Out-of-Domain Dependency Parsing for Dialects of Arabic: A Case Study

Noor Abo Mokh, Daniel Dakota, Sandra KüblerACL 2024

TL;DR

This paper tackles the challenge of dependency parsing for Arabic dialects, which lack annotated data. By training on Modern Standard Arabic (MSA) and adapting the model, they demonstrate that text genre differences pose a greater challenge than dialectal variations for parsing performance.

What Problem Does This Paper Address?

Syntactic parsing for Arabic dialects is a significant challenge due to the scarcity of syntactically annotated data. Existing Arabic treebanks are primarily in Modern Standard Arabic (MSA), which differs from spoken dialects in vocabulary, grammar, and style. This creates an "out-of-domain" problem: models trained on MSA perform poorly when applied to dialectal data. The paper aims to investigate methods to bridge this gap, focusing on improving parsing accuracy for dialectal Arabic by adapting MSA-trained models.

Methods

The authors employ a neural graph-based dependency parser (based on Dozat and Manning, 2017) and experiment with various domain adaptation techniques. These include:

  • Training on syntactically similar sentences: Selecting subsets of the MSA training data that are closer in length and potentially structure to the dialectal test data.
  • Removing sentential coordination: Investigating if removing MSA-specific sentential coordination structures from the training data improves performance on dialectal data, which tend to use phrasal coordination.
  • Using perplexity for similarity: Employing perplexity based on POS bigrams to select training sentences that are structurally similar to the test set, a common domain adaptation technique.
  • Adding in-domain MSA data: Incorporating a small, syntactically annotated MSA subset from a different corpus (MADAR) to the training data.
  • Using dialectal embeddings: Experimenting with different pre-trained Arabic BERT embeddings (araBERT, CAMEL BERT) and character embeddings to capture lexical differences.

Datasets

  • Training Data:
    • CATiB: A large MSA treebank (15,747 sentences) used as the primary source for out-of-domain training.
  • MADAR MSA: A smaller (2,000 sentences) syntactically annotated MSA subset from the MADAR corpus, used as "in-domain" training data.
  • Test Data:
    • MADAR Dialectal Corpus: Manually annotated test sets for four Arabic dialects (Egyptian, Gulf, Levantine, Maghrebi). For each dialect, 100 sentences were annotated for Idafa and 100 for coordination, resulting in a total of 800 parallel sentences.

Key Insights and Ideas

  • Genre Over Dialect: The most significant finding is that text genre differences between MSA and dialects have a greater negative impact on parsing performance than dialectal linguistic variations themselves. This suggests that adapting to the style of communication is crucial.
  • Idafa vs. Coordination: Idafa constructions, particularly prepositional ones, are challenging for MSA-trained parsers, likely due to their less standardized or different realization in dialects compared to MSA. Coordination also presents challenges, especially when MSA uses sentential coordination structures not common in dialects.
  • Sentence Length Matters: Shorter sentences in the training data (closer to dialectal test data) generally lead to improved parsing performance, indicating that syntactic complexity and sentence structure play a significant role.
  • Dialectal Embeddings Offer Limited Gains: While dialectal embeddings (CAMEL BERT) show some improvement, the gains are moderate, suggesting that lexical differences are not the primary bottleneck for out-of-domain parsing in this scenario.

Key Results

  • Out-of-Domain Baseline: Training on full CATiB (MSA) and testing on dialectal data resulted in a significant performance drop, with Unlabeled Attachment Scores (UAS) ranging from 55.1% to 57.5% and Labeled Attachment Scores (LAS) from 23.2% to 30.5%.
  • Best Performance: The highest overall results were achieved by adding the syntactically annotated MSA subset of MADAR to the training data, yielding UAS scores around 60-62% and LAS scores around 33-34%. This approach outperformed other adaptation strategies for Levantine, Egyptian, and Maghrebi dialects.
  • Idafa Performance: UAS for Idafa dependencies were considerably higher (84.6% to 91.6%) than LAS (38.6% to 54.2%), indicating that identifying the structure is easier than determining the dependency type.
  • Shortening Training Data: Training on shorter sentences from CATiB (up to 15 words) improved UAS and LAS by 2-3 points compared to using the full CATiB.
  • Removing Sentential Coordination: Removing sentences with sentential coordination from the training data led to improvements in both overall scores and coordination dependencies, and surprisingly, also for Idafa dependencies.

Limitations and Future Work

  • Limited Test Data: The study relies on a manually annotated test set of 800 sentences, which, while parallel across dialects, is still relatively small.
  • Focus on Two Phenomena: The research is limited to Idafa and coordination. Future work should extend to more syntactic phenomena.
  • Genre vs. Dialect Distinction: While the paper argues genre is more impactful, a more rigorous separation of these factors would be beneficial.
  • Future Directions: The authors plan to extend their study to more syntactic phenomena and investigate more traditional domain adaptation methods. They also aim to explore more traditional domain adaptation methods.

Summary updated

BibTeX
@inproceedings{mokh2024out,
  author = {Noor Abo Mokh and Daniel Dakota and Sandra Kübler},
  title = {{Out-of-Domain Dependency Parsing for Dialects of Arabic: A Case Study}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure