A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsing

Mark Anderson, Mathieu Dehouck, Carlos Gómez-RodríguezACL 2021

TL;DR

This paper investigates whether using predicted Universal Part-of-Speech (UPOS) tags as features can improve dependency parsing performance in low-resource settings. They find that predicted UPOS tags offer a modest benefit, especially when training data is scarce, but this benefit diminishes as more annotated data becomes available.

What Problem Does This Paper Address?

Dependency parsing, the task of analyzing the grammatical structure of sentences, is crucial for many Natural Language Processing (NLP) applications. However, building high-performing parsers typically requires large amounts of annotated data. For low-resource languages, where such data is scarce, achieving good parsing performance is a significant challenge. Part-of-speech (POS) tags are a common feature used to improve dependency parsers. However, in low-resource scenarios, obtaining accurate POS tags is also difficult. This paper explores the trade-off: can using predicted POS tags, even if imperfect, still help improve dependency parsing when gold-standard POS tags are unavailable or unreliable? They specifically focus on Universal POS (UPOS) tags, a standardized tagset, and investigate how the accuracy of these predicted tags and the amount of available training data influence parsing performance.

Methods

The authors conducted three main experiments:

  1. Real Low-Resource Treebanks: They evaluated the impact of predicted UPOS tags as features for biaffine dependency parsers on existing low-resource Universal Dependencies (UD) treebanks. They also tested a multi-task learning setup where UPOS tagging was an auxiliary task.
  2. Artificial Low-Resource Data: To control for data size, they created artificial low-resource treebanks by subsampling from high-resource languages. This allowed them to systematically vary the number of training sentences and evaluate the effect of different tagging accuracies.
  3. Data Augmentation: For very small treebanks (around 20 sentences), they employed a subtree swapping data augmentation technique to see if predicted UPOS tags could still be beneficial.

For predicting UPOS tags, they trained separate taggers and then used their predictions as input to the parsers. They also experimented with varying the accuracy of these predicted tags to understand its impact.

Datasets

The paper utilizes several datasets:

  • Real Low-Resource UD Treebanks:
    • Very Low Resource (< 50 sentences): Buryat BDT (bxr), Kazakh KTB (kk), Kurmanji MG (kmr), Livvi KKPP (olo), Upper Sorbian UFAL (hsb).
  • Low Resource (< 750 sentences): Belarusian HSE (be), Galician TreeGal (gl), Lithuanian HSE (lt), Marathi UFAL (mr), Old Russian RNC (orv), Tamil TTB (ta), Welsh CCG (cy).
  • Artificial Low-Resource Data: Created by subsampling from Indonesian GSD (id), Irish IDT (ga), Japanese GSD (ja), and Wolof WTB (wo).
  • Augmented Data: Generated from subsets of Kazakh, Kurmanji, and Upper Sorbian treebanks using subtree swapping.

Key Insights and Ideas

  • UPOS tags are helpful, but context matters: Predicted UPOS tags can provide a tangible boost to dependency parsing performance in low-resource settings, especially when the amount of fully annotated treebank data is very limited.
  • Diminishing returns: The benefit of using predicted UPOS tags decreases significantly as the size of the annotated treebank increases. Beyond a certain point, the added value is minimal.
  • Tagging accuracy is a trade-off: While higher tagging accuracy generally leads to better parsing, even moderately accurate predicted UPOS tags can be useful in low-resource scenarios, suggesting that perfect taggers are not always a prerequisite.
  • Data augmentation can amplify benefits: For extremely small treebanks, data augmentation techniques can enhance performance, and predicted UPOS tags can further contribute to this improvement, even when derived from low-accuracy taggers.
  • Multi-task learning's mixed results: The multi-task learning approach (where tagging is an auxiliary task) showed mixed results, sometimes helping and sometimes hindering performance, suggesting it's not a universally superior solution for this problem.

Key Results

  • For real low-resource treebanks with less than 50 sentences, using predicted UPOS tags increased performance, though the overall scores remained very low.
  • For treebanks with less than 750 sentences, predicted UPOS tags provided an average increase of about 1.2% in LAS (Labeled Attachment Score) over parsers without tags, with an average tagging accuracy of 85.89%. This suggests that in lower-resource settings, the required tagging accuracy might be lower than in high-resource settings.
  • When using artificial low-resource data, if sufficient data was available (e.g., 232 or 541 samples), using UPOS tags offered no improvement. However, with fewer samples (e.g., 100), there was a slight upward trend when using tags from more accurate taggers.
  • Augmented data consistently improved performance on very small treebanks. For these cases, predicted UPOS tags could still be beneficial, but there was no clear trend showing the impact of UPOS accuracy.

Limitations and Future Work

  • Pretrained Language Models: The authors acknowledge that their experiments did not consider the potential impact of pretrained multilingual language models (LMs), which might offer similar or even better benefits than explicit POS tags. They suggest future work could investigate this interaction.
  • UD Tagset Informativeness: One reviewer suggested that the UD tagset itself might not be sufficiently informative for some low-resource languages. The authors agree this is a possibility but emphasize that the low accuracy of taggers is likely the primary contributing factor.
  • Implicit Information in LMs: Future research could explore whether the implicit information encoded in LMs is more useful than explicit POS tag information.

Summary updated

BibTeX
@inproceedings{anderson2021falta,
  author = {Mark Anderson and Mathieu Dehouck and Carlos Gómez-Rodríguez},
  title = {{A Falta de Pan, Buenas Son Tortas: The Efficacy of Predicted UPOS Tags for Low Resource UD Parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure