Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resource

Antonios Anastasopoulos, Marika Lekakou, Josep Quer, Eleni Zimianiti, Justin DeBenedetto, David Chiang

TL;DR

This paper introduces a new parallel Griko-Italian corpus and evaluates various Part-of-Speech (POS) tagging methods on this endangered language. They find that combining semi-supervised learning with cross-lingual projections offers the best performance for limited data, while active learning eventually makes simpler models competitive.

What Problem Does This Paper Address?

Natural Language Processing (NLP) tools are overwhelmingly developed for high-resource languages, leaving a vast majority of the world's languages, especially endangered ones, underserved. Endangered languages present unique challenges due to a lack of standardized orthography and limited linguistic resources. This paper addresses the critical need for developing NLP tools for such languages by focusing on Griko, a Greek dialect spoken in Southern Italy. Specifically, they tackle the problem of POS tagging, a fundamental task for many NLP applications, in this low-resource and challenging setting.

Methods

The authors explore several POS tagging strategies:

  • Traditional Models: They start with established methods like a feature-based cross-entropy tagger and a Conditional Random Field (CRF) tagger.
  • Semi-Supervised Learning: They leverage monolingual Griko data to improve tagging accuracy, following approaches like Garrette and Baldridge (2013).
  • Cross-Lingual Transfer: They utilize the Italian translations to project POS tags from Italian to Griko. This involves creating word alignments between the parallel sentences and mapping Italian tags to a universal tagset.
  • Combined Approaches: A key contribution is the combination of semi-supervised learning with cross-lingual tag projection, showing improved performance.
  • Active Learning: They implement an active learning scheme where linguists iteratively annotate data, and the tagger is retrained. This allows them to study how performance evolves as more annotated data becomes available.

Datasets

  • Griko-Italian Parallel Corpus: The authors introduce a new resource consisting of 114 Griko narratives with sentence-level Italian translations. This corpus was collected from early 20th-century manuscripts.
  • UoI Corpus: They also reference an existing Griko corpus (Lekakou et al., 2013) which contains about 20 minutes of speech with Italian translations, morphosyntactic tags, and glossing.
  • Test Set: A subset of 10 narratives from their new corpus was hand-annotated by linguists to serve as the gold standard test set.

Key Insights and Ideas

  • The Power of Combination: For very limited annotated data, combining semi-supervised learning (using monolingual data) with cross-lingual tag projection significantly boosts POS tagging accuracy. This suggests that leveraging both internal language structure and external linguistic knowledge is crucial.
  • Active Learning's Effectiveness: Active learning proves highly effective, leading to substantial accuracy gains as more data is annotated. This is particularly important for endangered languages where manual annotation is costly.
  • Simpler Models Shine with More Data: As the amount of annotated data grows through active learning, simpler, feature-based CRF models can outperform more complex methods, even negating the need for cross-lingual projections. This highlights the trade-off between model complexity and data availability.
  • Challenges of Endangered Languages: The paper underscores the difficulties posed by non-standard orthography and word fusion in Griko, requiring novel annotation strategies (e.g., multi-tagging fused words).

Key Results

  • Initial Performance: With only 360 annotated sentences, the best method (combining cross-lingual projections and monolingual data) achieved 72.9% accuracy.
  • Active Learning Gains: Through active learning, accuracy improved by over 21 percentage points.
  • Final Performance: With sufficient data (over 94% accuracy on the last narrative), a simpler feature-based CRF model achieved the highest scores.
  • Cross-lingual Projections: While beneficial initially, cross-lingual tag projections sometimes hurt performance when more data is available, suggesting they can introduce noise.

Limitations and Future Work

  • Orthographic Variability: The lack of a consistent orthography in Griko poses ongoing challenges for tokenization and annotation.
  • Limited Test Set: While the test set is hand-annotated, it represents only 10 narratives.
  • Future Work: The authors plan to:
  • Expand POS annotations to the entire corpus.
  • Enrich the corpus with morphological tags.
  • Use statistical methods to study Griko grammar and syntax.
  • Contribute the corpus to the Universal Dependencies treebanks.
  • Explore word-level active learning for more efficient annotation.

Summary updated

Abstract

Most work on part-of-speech (POS) tagging is focused on high resource languages, or examines low-resource and active learning settings through simulated studies. We evaluate POS tagging techniques on an actual endangered language, Griko. We present a resource that contains 114 narratives in Griko, along with sentence-level translations in Italian, and provides gold annotations for the test set. Based on a previously collected small corpus, we investigate several traditional methods, as well as methods that take advantage of monolingual data or project cross-lingual POS tags. We show that the combination of a semi-supervised method with cross-lingual transfer is more appropriate for this extremely challenging setting, with the best tagger achieving an accuracy of 72.9%. With an applied active learning scheme, which we use to collect sentence-level annotations over the test set, we achieve improvements of more than 21 percentage points.
BibTeX
@inproceedings{anastasopoulos2018part,
  author = {Antonios Anastasopoulos and Marika Lekakou and Josep Quer and Eleni Zimianiti and Justin DeBenedetto and David Chiang},
  title = {{Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resource}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=Ua7M7Hek3do},
}
Figure