A survey of part-of-speech tagging approaches applied to K’iche’

Francis Tyers, Nick HowellNAACL 2021

TL;DR

This paper surveys the performance of popular neural part-of-speech (POS) taggers on K'iche', a Mayan language, achieving impressive F1 scores over 93% for lemmatization, POS tagging, and morphological feature assignment. The study then explores cross-language tagging on related Mayan languages, finding that morphological diversity is a key factor limiting performance, suggesting directions for future annotation efforts.

What Problem Does This Paper Address?

Part-of-speech (POS) tagging is a fundamental task in computational linguistics, crucial for many downstream NLP applications. However, for low-resource languages, particularly indigenous languages like those of the Mayan family, annotated data is scarce, making it challenging to apply and evaluate state-of-the-art NLP models. This paper addresses this gap by evaluating existing, popular POS tagging systems on K'iche', a Mayan language, and then investigating the feasibility of cross-language tagging to other Mayan languages (Kaqchikel and Uspanteko) with even less data. The goal is to understand the performance of current models and identify factors that hinder or facilitate their application to related, under-resourced languages.

Methods

The authors employ a survey-style approach, evaluating several popular Universal Dependencies (UD) ecosystem POS taggers. They use a curated K'iche' corpus of 1435 annotated sentences (approximately 10,000 tokens) for training and evaluation. For K'iche' evaluation, they perform 10-fold cross-validation to obtain mean and standard deviation of F1 scores for lemmatization, UPOS (Universal Part-of-Speech tags), and UFeats (Universal Morphological Features). They also analyze resource consumption (training time and model size).

To investigate cross-language tagging, they use the best-performing K'iche'-trained models to tag Kaqchikel and Uspanteko, using small, newly annotated test sets for these languages. A convergence study is conducted to model the relationship between training data size and performance, using a formula to estimate asymptotic performance and the data required to reach specific performance targets.

Datasets

  • K'iche' Corpus: A newly annotated corpus of 1435 K'iche' sentences, comprising approximately 10,000 tokens. This corpus was annotated according to Universal Dependencies (UD) guidelines.
  • Kaqchikel Corpus: A small test corpus of 157 sentences (1091 tokens), annotated for POS, lemmas, and morphological features.
  • Uspanteko Corpus: A small test corpus of 160 sentences (1171 tokens), annotated for POS, lemmas, and morphological features.

Key Insights and Ideas

  • High Performance on K'iche': State-of-the-art neural POS taggers achieve remarkably high performance on K'iche', even with a relatively small corpus. This suggests that UD-trained models are robust and can generalize well to morphologically rich languages when sufficient data is available.
  • Morphological Diversity as a Bottleneck: The significant drop in performance when applying K'iche'-trained models to Kaqchikel and especially Uspanteko points to morphological diversity as a primary challenge for cross-language tagging. Differences in feature sets and their realization between languages are likely the limiting factors.
  • UDify's Limitations: While UDify, a BERT-based multilingual model, shows promise, it struggles to overcome biases from its pre-training on non-Mayan languages. This highlights the need for more linguistically diverse pre-training data or specialized models for Mayan languages.
  • Resource Efficiency of UDPipe: The original UDPipe model demonstrates impressive resource efficiency, achieving performance close to UDPipe 2 with a fraction of the training time and model size.
  • Convergence Study Insights: The convergence study reveals that while performance improves with more data, the gains diminish exponentially. This provides a framework for estimating the data needed to reach desired performance levels and suggests that targeted annotation of specific linguistic features could be more effective than simply increasing data size.

Key Results

  • K'iche' Performance: The best-performing system (UDPipe 2) achieved F1 scores of 95.4±1.9% for lemmatization, 97.4±2.2% for UPOS, and 95.7±2.1% for UFeats on the K'iche' corpus.
  • Cross-Language Performance:
    • Kaqchikel: F1 scores ranged from 63% to 85% for UPOS, features, and lemmas.
  • Uspanteko: F1 scores were more modest, ranging from 60% to 71%.
  • UDify-FT vs. Original UDify: The UDify model fine-tuned on K'iche' (UDify-FT) significantly outperformed the original UDify model on Kaqchikel and Uspanteko, though UDPipe 2 performed comparably to UDify-FT on Kaqchikel.
  • Resource Consumption: UDPipe was the most resource-efficient, using significantly less energy and model size compared to UDPipe 2 and UDify.

Limitations and Future Work

  • Small Test Corpora: The test corpora for Kaqchikel and Uspanteko are small and may not fully represent the diversity of these languages. This limits the generalizability of the cross-language findings.
  • Morphological Diversity: The authors identify the relative diversity of morphological features as a plausible explanation for the limiting factors in cross-language tagging performance.
  • Future Work Suggestions:
    • Targeted Annotation: Future work should focus on annotating more data that includes linguistic features present in the target languages (Kaqchikel, Uspanteko) but underrepresented in the K'iche' training corpus.
  • Improving UDify: Investigating whether training UDify on data with a better representation of American languages could improve its performance.
  • Cross-Language Tagging as a Path Forward: The study suggests that cross-language tagging is a viable strategy for increasing the availability of POS annotation for Mayan languages.

Summary updated

BibTeX
@inproceedings{tyers2021survey,
  author = {Francis Tyers and Nick Howell},
  title = {{A survey of part-of-speech tagging approaches applied to K’iche’}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
}
Figure