Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resource

Antonis Anastasopoulos, Marika Lekakou, Josep Quer, Eleni Zimianiti, Justin DeBenedetto, David Chiang

Abstract

Most work on part-of-speech (POS) tagging is focused on high-resource languages, or examines low-resource and active learning settings through simulated studies. We evaluate POS tagging techniques on an actual endangered language, Griko. We present a resource that contains 114 narratives in Griko, along with sentence-level translations in Italian, and provides gold annotations for the test set. Based on a previously collected small corpus, we investigate several traditional methods, as well as methods that take advantage of monolingual data or project cross-lingual POS tags. We show that the combination of a semi-supervised method with cross-lingual transfer is more appropriate for this extremely challenging setting, with the best tagger achieving an accuracy of 72.9%. With an applied active learning scheme, which we use to collect sentence-level annotations over the test set, we achieve improvements of more than 21 percentage points.

Introduction

Natural Language Processing (NLP) research predominantly operates within a "high-resource" bubble, prioritizing languages like English or Mandarin where data is abundant. When low-resource settings are explored, they are often simulated by artificially restricting large datasets. This paper addresses the "messy" reality of Griko, a seriously endangered Greek dialect spoken by fewer than 20,000 people in the Salento region of Southern Italy.

Griko presents unique computational challenges that standard models often fail to capture:

  1. Orthographic Variance: The lack of a standard spelling system leads to significant noise in string matching.
  2. Morphological Fusion: Multiple parts of speech often fuse into a single token (e.g., "to the" becoming one word), necessitating complex multi-tagging.
  3. Code-Switching: Speakers frequently oscillate between Griko, Italian, and the local Salentino dialect.

Methodology and Data

The researchers introduced a significant new resource: a parallel corpus of 114 narratives (approx. 10,000 sentences) paired with Italian translations. A gold-standard test set of 10 narratives was hand-annotated by linguists to provide a reliable benchmark.

To evaluate performance, several architectures were compared:

  • Baselines: Stanford Log-linear tagger, a basic Conditional Random Field (CRF), and a neural bi-LSTM.
  • Enhanced CRF (crf-mod): Augmented with prefix/suffix features to capture Griko's rich morphology.
  • Semi-Supervised Learning (G&B): Utilizing the Garrette & Baldridge framework to leverage unannotated monolingual data via label propagation.
  • Cross-Lingual Projection: Projecting POS tags from the high-resource Italian translations onto aligned Griko words.

Key Results and Insights

The study revealed that in extreme low-resource scenarios, hybrid approaches are essential. The best initial performance was achieved by combining semi-supervised learning with cross-lingual transfer from Italian.

POS tagging accuracy across different models and data settings.

As shown in the results table, the G&B model with cross-lingual projection (+clp) reached an accuracy of 72.92%. Interestingly, as the training set grew to roughly 800 sentences, the well-tuned crf-mod began to outperform more complex semi-supervised models, suggesting that feature engineering remains highly effective when data is scarce but growing.

The Impact of Active Learning

To bridge the gap to high-accuracy tagging, the authors implemented an active learning scheme. By iteratively selecting narratives for human annotation and retraining the models, they simulated a realistic workflow for field linguists.

Iterative accuracy improvements through active learning.

The results were dramatic: accuracy improved by over 21 percentage points, eventually reaching 94.17%. This demonstrates that even a small amount of targeted human intervention can yield massive gains in low-resource NLP settings.

Conclusion

This work provides both a valuable resource for the Griko language and a roadmap for applying NLP to other endangered varieties. While challenges like code-switching and non-standard orthography remain, the success of cross-lingual projection and active learning offers a path forward for preserving linguistic diversity through technology.

Paper Citation: Anastasopoulos, A., Lekakou, M., Quer, J., Zimianiti, E., DeBenedetto, J., & Chiang, D. (2018). Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resource. arXiv:1806.03757 [cs.CL].

Summary updated

Abstract

Most work on part-of-speech (POS) tagging is focused on high resource languages, or examines low-resource and active learning settings through simulated studies. We evaluate POS tagging techniques on an actual endangered language, Griko. We present a resource that contains 114 narratives in Griko, along with sentence-level translations in Italian, and provides gold annotations for the test set. Based on a previously collected small corpus, we investigate several traditional methods, as well as methods that take advantage of monolingual data or project cross-lingual POS tags. We show that the combination of a semi-supervised method with cross-lingual transfer is more appropriate for this extremely challenging setting, with the best tagger achieving an accuracy of 72.9%. With an applied active learning scheme, which we use to collect sentence-level annotations over the test set, we achieve improvements of more than 21 percentage points.
BibTeX
@inproceedings{anastasopoulos2018part,
  author = {Antonis Anastasopoulos and Marika Lekakou and Josep Quer and Eleni Zimianiti and Justin DeBenedetto and David Chiang},
  title = {{Part-of-Speech Tagging on an Endangered Language: a Parallel Griko-Italian Resource}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
  url = {https://openreview.net/forum?id=Ua7M7Hek3do},
}
Figure