Cross-lingual Annotation Projection Is Effective for Neural Part-of-Speech Tagging

Matthias Huck, Diana Dutka, Alexander Fraser2019 · DOI

TL;DR

This paper investigates effective strategies for Part-of-Speech (POS) tagging in Ukrainian, a low-resource language. They demonstrate that cross-lingual annotation projection from a related language (Russian) is a highly effective method for the zero-resource scenario, achieving competitive results even without any Ukrainian training data.

What Problem Does This Paper Address?

POS tagging is a fundamental task in Natural Language Processing (NLP), crucial for understanding sentence structure and meaning. However, creating high-quality POS-tagged datasets is labor-intensive and expensive, making it challenging for the vast majority of the world's languages, which are considered "low-resource." This paper focuses on addressing the extreme cases: the "zero-resource" scenario (no labeled data) and the "low-resource" scenario (limited labeled data) for Ukrainian. They aim to determine how effectively neural models can perform POS tagging under these constraints, comparing various techniques to leverage available resources, particularly a related high-resource language like Russian.

Methods

The authors employ a state-of-the-art Bidirectional Long Short-Term Memory (BLSTM) neural network model for sequence tagging. They compare four main strategies:

  • Zero-shot Tagging: A model trained on a high-resource language (Russian) is directly applied to the target language (Ukrainian) without any further training on Ukrainian data.
  • Cross-lingual Annotation Projection: This involves using a parallel corpus between Russian and Ukrainian. POS tags from the Russian side are projected onto the Ukrainian side using word alignments, creating synthetic training data for Ukrainian.
  • Self-Training: An initial model is trained on available labeled data (or projected data), then used to label unlabeled data. This newly labeled data is added to the training set to retrain the model, iteratively improving performance.
  • Multilingual Learning: Training data from a related language (Russian) is combined with the limited Ukrainian training data to enlarge the training set and improve generalization.

They also explore the impact of stemming (reducing words to their root form) on vocabulary overlap and tagging performance, especially for morphologically rich languages like Ukrainian and Russian.

Datasets

  • Ukrainian Universal Dependencies (UD) Corpus: A small, manually annotated dataset for Ukrainian, used for low-resource training and evaluation.
  • Russian Universal Dependencies (UD) Corpus: A larger, manually annotated dataset for Russian, used for training Russian models and as a source for cross-lingual transfer.
  • Russian-Ukrainian Parallel Corpus: A corpus of parallel news texts used for cross-lingual annotation projection.

Key Insights and Ideas

  • Cross-lingual projection shines in zero-resource: The most significant finding is that cross-lingual annotation projection from Russian to Ukrainian is remarkably effective, achieving accuracies close to supervised low-resource methods even when no Ukrainian training data is available. This suggests that leveraging parallel corpora and linguistic similarity can overcome the lack of direct supervision.
  • Zero-shot tagging is insufficient: Directly applying a Russian-trained model to Ukrainian (zero-shot) yields poor results, indicating that simple transfer without projection or adaptation is not enough, even for related languages.
  • Stemming helps, especially for OOVs: Reducing words to their stems increases vocabulary overlap between Ukrainian and Russian and significantly improves the handling of Out-Of-Vocabulary (OOV) words, particularly in zero-shot and multilingual settings.
  • Self-training offers marginal gains: While self-training can augment training data, its effectiveness is limited, likely due to domain mismatch or the inherent noise in synthetically generated labels.
  • Pre-trained embeddings are crucial: Using pre-trained word embeddings significantly boosts the performance of BLSTM taggers compared to randomly initialized embeddings, highlighting their importance in capturing semantic and syntactic information.

Key Results

  • Cross-lingual Transfer: Achieved an accuracy of 84.4% on the Ukrainian test set in the zero-resource scenario.
  • Multilingual Learning (with stemmed data): Achieved the highest overall accuracy of 87.3% for the low-resource scenario, by training on a combined set of stemmed Ukrainian and Russian data.
  • Low-resource BLSTM + PE: Reached 85.4% accuracy using pre-trained embeddings on Ukrainian data alone.
  • Zero-shot Tagging: Achieved only 51.5% accuracy, demonstrating its inadequacy.
  • Self-Training: Improved accuracy from 85.4% to 86.2%, a modest gain.

Limitations and Future Work

  • Limitations: The authors acknowledge that zero-shot tagging's poor performance might be due to the relatively small amount of shared vocabulary and certain grammatical differences between Ukrainian and Russian. The effectiveness of cross-lingual transfer is also dependent on the quality of word alignment.
  • Future Work: The paper suggests that further improvements in cross-lingual transfer could be achieved by tailoring word alignment more specifically to the task. Combining stemmed words with pre-trained stem embeddings is also proposed as a promising avenue for future research. For low-resource scenarios, multilingual learning is recommended, and for zero-resource, cross-lingual projection is strongly advised when parallel corpora are available.

Summary updated

Abstract

We tackle the important task of part-of-speech tagging using a neural model in the zero-resource scenario, where we have no access to gold-standard POS training data. We compare this scenario with the low-resource scenario, where we have access to a small amount of gold-standard POS training data. Our experiments focus on Ukrainian as a representative of under-resourced languages. Russian is highly related to Ukrainian, so we exploit gold-standard Russian POS tags. We consider four techniques to perform Ukrainian POS tagging: zero-shot tagging and cross-lingual annotation projection (for the zero-resource scenario), and compare these with self-training and multilingual learning (for the low-resource scenario). We find that cross-lingual annotation projection works particularly well in the zero-resource scenario.
BibTeX
@misc{huck2019cross,
  author = {Matthias Huck and Diana Dutka and Alexander Fraser},
  title = {{Cross-lingual Annotation Projection Is Effective for Neural Part-of-Speech Tagging}},
  year = {2019},
  doi = {10.18653/v1/w19-1425},
}
Figure