Weakly Supervised POS Taggers Perform Poorly on Truly Low-Resource Languages

Katharina Kann, Ophélie Lacroix, Anders Søgaard

TL;DR

Existing "weakly supervised" part-of-speech (POS) taggers report near-supervised accuracy, but they are almost always evaluated on simulated low-resource setups using clean, high-coverage dictionaries. When evaluated on 15 truly low-resource languages with realistic data constraints, even state-of-the-art architectures fail to reach 50% accuracy—revealing a substantial reality gap in low-resource NLP.

What Problem Does This Paper Address?

Part-of-speech (POS) tagging is a foundational pipeline task in natural language processing. While supervised taggers regularly score above 97% accuracy on high-resource languages, curating annotated treebanks requires trained linguists and significant funding—luxuries unavailable for thousands of the world’s languages.

To bridge this gap, prior research developed weakly supervised methods using cross-lingual transfer, parallel corpora, or type-level dictionaries (e.g., Wiktionary). Many of these papers claimed tagging accuracies topping 80%. However, the authors argue this is an illusion: previous evaluations relied on high-resource proxies (e.g., German or Spanish treated as "low-resource") and artificially clean, dense lexicons. This paper asks: How well do weakly supervised and unsupervised POS taggers actually perform when applied to truly low-resource languages with noisy, sparse real-world resources?

Methods

The authors investigate completely zero-annotation target settings (no labeled gold sentences in the target language). They benchmark several baselines and propose a regularized neural transfer model:

  • Baselines:
  • CHR11: A fully unsupervised Bayesian multinomial mixture model that induces word clusters, which are mapped to POS tags via small dictionaries.
  • GAR13: A Maximum Entropy Markov Model trained on noisily labeled text expanded via label propagation.
  • PLA16: A hierarchical bi-LSTM trained via multi-task learning (MTL) with an auxiliary word-level language modeling objective (predicting next-word log frequency).
  • Proposed Cross-Lingual Labeling Strategies: Using English as the source and bilingual word-level dictionaries (from PanLex Swadesh and Wikidata), sentences from target-language Wikipedia are labeled via:
  • Frequency-Based Annotation (FREQ): Tokens are assigned the most frequent POS tag associated with their English translations in gold English treebanks.
  • Ambiguous Annotation (AMB): Tokens are assigned all candidate tags found across translations and monolingual dictionaries; training treats the model's highest-probability candidate tag as the gold label.
  • Character-Level Seq2Seq Multi-Task Regularization (AMB+AE / FREQ+AE): Hierarchical LSTMs are expressive and prone to severe overfitting on small, noisy corpora. To fix this, the authors add an auxiliary character-level sequence-to-sequence autoencoder (reproducing an input word character-by-character) sharing the subword character-biLSTM encoder with the tagger.

Datasets

  • Target Languages: 15 truly low-resource languages spanning 6 language families: Amharic, Belarusian, Breton, Buryat, Faroese, Upper Sorbian, Armenian, Kazakh, Kurmanji, Lithuanian, Marathi, Maltese, Tamil, Telugu, and Tagalog. Evaluation uses test splits from Universal Dependencies (UD) v2.1.
  • High-Resource Comparison Set: German, Spanish, Italian, Portuguese, and Swedish evaluated on the PUD treebanks.
  • External Resources: Unlabeled Wikipedia dumps, Polyglot word embeddings, bilingual wordlists from PanLex Swadesh and Wikidata, and morphological tags from UniMorph (available for only 3 of the 15 languages).

Key Insights and Ideas

  • The "Low-Resource" Illusion: Prior successes in weakly supervised tagging were driven by dictionary coverage and dataset cleanliness rather than robust transfer algorithms. When tested on actual low-resource conditions, performance drops dramatically.
  • Word-Level Auxiliary Tasks Hurt, Character Tasks Help: While language-modeling auxiliary losses (PLA16) help high-resource taggers, they hurt performance in low-resource environments because word frequencies cannot be estimated accurately from tiny corpora. Conversely, character autoencoding (AE) successfully regularizes character embeddings against noisy supervision.
  • Severe Label Imbalance from Dictionaries: Naive dictionary projection produces skewed tag distributions. In FREQ, punctuation and verbs were dramatically over-represented compared to true test distributions, while nouns were heavily under-represented.

Key Results

  • All models struggle: Across all 15 languages, no system achieved even 50% macro-average accuracy.
  • Baseline performance: Unsupervised clustering (CHR11) averaged 34.40%, while the strongest existing baseline (GAR13) achieved 39.11%. PLA16 reached 38.99%.
  • Proposed approaches: The single-task neural models scored 41.06% (FREQ) and 41.38% (AMB). Adding character-level autoencoding yielded the top overall performance at 42.25% (AMB+AE).
  • High-Resource Sanity Check: When evaluated on high-resource European languages using standard, high-quality Wiktionary annotations ("Wiki-ly"), AMB jumped to 72% and FREQ reached 77% (compared to 43% and 50% when restricted to low-resource-style dictionaries).

Limitations and Future Work

  • English Pivot Bias: The authors used English as the single source language for cross-lingual transfer due to dictionary availability, rather than choosing linguistically related, typological donor languages.
  • Contextualized Embeddings Omitted: Pretrained language models (like mBERT or XLM-R) were excluded due to poor language identification, heavy spelling variation, and lack of pretraining data for genuine low-resource languages.
  • Future Direction: The authors emphasize that minor architectural tweaks on top of existing supervised frameworks will not bridge this performance gap; truly low-resource POS tagging requires radically different paradigms for distant supervision and syntax induction.

Summary updated

Abstract

Part-of-speech (POS) taggers for low-resource languages which are exclusively based on various forms of weak supervision – e.g., cross-lingual transfer, type-level supervision, or a combination thereof – have been reported to perform almost as well as supervised ones. However, weakly supervised POS taggers are commonly only evaluated on languages that are very different from truly low-resource languages, and the taggers use sources of information, like high-coverage and almost error-free dictionaries, which are likely not available for resource-poor languages. We train and evaluate state-of-the-art weakly supervised POS taggers for a typologically diverse set of 15 truly low-resource languages. On these languages, given a realistic amount of resources, even our best model gets only less than half of the words right. Our results highlight the need for new and different approaches to POS tagging for truly low-resource languages.
BibTeX
@inproceedings{kann2020weakly,
  author = {Katharina Kann and Ophélie Lacroix and Anders Søgaard},
  title = {{Weakly Supervised POS Taggers Perform Poorly on Truly Low-Resource Languages}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=Qiad7OuXWDy},
}

Loading related content...

Figure