New syntactic insights for automated Wolof Universal Dependency parsing

Bill DyerACL 2022

TL;DR

This paper explores how adapting Universal Dependencies (UD) annotation schemes to better reflect Wolof's unique syntax can significantly improve automated parsing accuracy. By treating relative clause pronouns as determiners and reclassifying copulas based on their function, the authors achieved notable accuracy gains on existing UD treebanks, demonstrating the importance of language-specific linguistic insights for low-resource languages.

What Problem Does This Paper Address?

Natural Language Processing (NLP) often relies on universal frameworks like Universal Dependencies (UD) to standardize linguistic annotation across languages. However, applying these universal schemes to low-resource languages, especially those with typologically different structures from widely studied languages like English, can be challenging. Wolof, a West African language with limited digital resources and existing UD treebanks, exemplifies this problem. Existing UD parsers, trained on data that might force Wolof's syntax into a mold designed for other languages, may not perform optimally. This paper aims to improve Wolof UD parsing by investigating how specific syntactic and morphological analyses, informed by linguistic insights into Wolof, can enhance parser performance.

Methods

The core of this research involves creating edited versions of the Wolof UD treebank to test specific hypotheses about its syntax. Two primary editing strategies were employed:

  1. DEF Tag for Relative Pronouns: The researchers hypothesized that relative clause pronouns and definite determiners in Wolof share functional similarities. They proposed reclassifying relative clause pronouns from the PRON tag to a new DEF tag, treating them as determiners (det) or complementizers (mark). This was applied to both headed and headless relative clauses, aiming for a unified dependency structure.
  2. Relabeling Copulas: The study questioned the direct application of English-centric copula analysis to Wolof. Instead, they proposed reclassifying words tagged as COP (copula) in the Wolof-specific tagset to INFL (inflectional particle) or AUX (auxiliary verb) in the universal tagset, based on their functional role (e.g., indicating aspect or focus) rather than a direct copular function.

These edited treebanks were then used to train and evaluate two different parser pipelines: spaCy and TreeTagger+MaltParser. The performance of models trained on these edited datasets was compared against a baseline model trained on the original, unedited Wolof UD treebank.

Datasets

The study utilized the existing Wolof UD Treebank, which consists of 42,832 tokens across 2,107 sentences. This data was originally sourced from four online Wolof resources: the Organisation Sénégalaise d'Appui au Développement (OSAD) website, Wolof Online, Wolof Wikipedia, and the news site Xibaaryi. The data was divided into training, development, and test sets by the original annotator. The researchers created three edited versions of this dataset for their experiments:

  • One with relative pronouns tagged as DEF and assigned the det relation.
  • One with relative pronouns tagged as DEF and assigned the mark relation.
  • One with copulas relabeled as INFL or AUX.

A combined model using the best DEF tag and relabeled copulas was also evaluated.

Key Insights and Ideas

  • Language-Specific Syntax Matters: The study strongly supports the idea that adapting annotation schemes to a language's unique syntactic features, rather than forcing it into a universal mold, leads to improved parsing accuracy.
  • Unified Relative Clause Structure: Treating headless relative clauses with the same dependency structure as headed ones, by reclassifying their pronouns, simplifies the annotation and improves parser performance.
  • Wolof Copulas are Not English Copulas: The direct mapping of English copula analysis to Wolof is problematic. Reclassifying these elements based on their functional role (aspect, focus) rather than a strict copular definition yields better results.
  • Relative Pronoun Ambiguity: The research found that treating relative clause pronouns as determiners (det) led to more accurate parsing than treating them as complementizers (mark), contradicting one of the initial hypotheses but providing a valuable insight.
  • Low-Resource Language Gains: Even small percentage improvements in accuracy are significant for low-resource languages like Wolof, where data is scarce and annotation efforts are costly.

Key Results

The edited treebanks consistently led to improved parsing accuracy compared to the baseline model trained on the original Wolof UD Treebank.

  • spaCy Pipeline:
    • The DEF tag model with relative pronouns as det improved UD label accuracy by 1.5% and universal dependency accuracy by 2.5%.
  • The DEF tag model with relative pronouns as mark improved UD label accuracy by 1.4% and universal dependency accuracy by 2.8%.
  • Relabeling copulas improved spaCy UD labels by 1% and universal dependencies by 0.3% (with a maximum of 0.6% improvement in spaCy universal dependencies).
  • Combining the best DEF tag model and the relabeled copula model resulted in a 1.6% increase in spaCy UD label accuracy and a 0.3% increase in spaCy universal dependency accuracy.
  • TreeTagger+MaltParser Pipeline:
    • The DEF tag model with relative pronouns as det showed a 2.2% increase in Malt UD labels and a 2.5% increase in Malt universal dependencies.
  • The DEF tag model with relative pronouns as mark improved Malt UD labels by 1.3% and Malt universal dependencies by 2.8%.
  • Relabeling copulas increased Malt UD label accuracy by 0.8% and Malt universal dependencies by 0.3%.
  • The combined model achieved a 3.4% increase in Malt UD label accuracy and a 3.3% increase in Malt universal dependencies.

Qualitatively, the relabeling of relative clause pronouns as determiners led to an increase in precision, recall, and F1-score for these elements, as they were no longer confused with subjects or objects. The relabeling of copulas also resulted in improved precision, recall, and F1-scores for auxiliary verbs.

Limitations and Future Work

  • Limitations: The study acknowledges that while the accuracy improvements are significant for a low-resource language, the absolute accuracy achieved by the best model (78% recall, 78% precision, 78% F-score) is comparable to, but not significantly higher than, a previously developed Wolof-specific lexical functional grammar parser (67% recall, 93% precision, 78% F-score). The authors suggest that combining their edited treebanks with Dione's parser could lead to even greater accuracy.
  • Future Work: The paper suggests that future parsers could benefit from combining the annotation edits proposed here with Dione's existing Wolof-specific parser. The improvements made to parsing Wolof provide a roadmap for future annotation and parsing efforts for other underrepresented African languages within the UD framework. The authors also note that the minimalist syntax framework offers a way to maintain a unified structure while allowing for null elements, which can inform future annotation strategies.

Summary updated

BibTeX
@inproceedings{dyer2022new,
  author = {Bill Dyer},
  title = {{New syntactic insights for automated Wolof Universal Dependency parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}

Loading related content...

Figure