TL;DR
Existing "weakly supervised" part-of-speech (POS) taggers report near-supervised accuracy, but they are almost always evaluated on simulated low-resource setups using clean, high-coverage dictionaries. When evaluated on 15 truly low-resource languages with realistic data constraints, even state-of-the-art architectures fail to reach 50% accuracy—revealing a substantial reality gap in low-resource NLP.
What Problem Does This Paper Address?
Part-of-speech (POS) tagging is a foundational pipeline task in natural language processing. While supervised taggers regularly score above 97% accuracy on high-resource languages, curating annotated treebanks requires trained linguists and significant funding—luxuries unavailable for thousands of the world’s languages.
To bridge this gap, prior research developed weakly supervised methods using cross-lingual transfer, parallel corpora, or type-level dictionaries (e.g., Wiktionary). Many of these papers claimed tagging accuracies topping 80%. However, the authors argue this is an illusion: previous evaluations relied on high-resource proxies (e.g., German or Spanish treated as "low-resource") and artificially clean, dense lexicons. This paper asks: How well do weakly supervised and unsupervised POS taggers actually perform when applied to truly low-resource languages with noisy, sparse real-world resources?
Methods
The authors investigate completely zero-annotation target settings (no labeled gold sentences in the target language). They benchmark several baselines and propose a regularized neural transfer model:
- Baselines:
- CHR11: A fully unsupervised Bayesian multinomial mixture model that induces word clusters, which are mapped to POS tags via small dictionaries.
- GAR13: A Maximum Entropy Markov Model trained on noisily labeled text expanded via label propagation.
- PLA16: A hierarchical bi-LSTM trained via multi-task learning (MTL) with an auxiliary word-level language modeling objective (predicting next-word log frequency).
- Proposed Cross-Lingual Labeling Strategies: Using English as the source and bilingual word-level dictionaries (from PanLex Swadesh and Wikidata), sentences from target-language Wikipedia are labeled via:
- Frequency-Based Annotation (FREQ): Tokens are assigned the most frequent POS tag associated with their English translations in gold English treebanks.
- Ambiguous Annotation (AMB): Tokens are assigned all candidate tags found across translations and monolingual dictionaries; training treats the model's highest-probability candidate tag as the gold label.
- Character-Level Seq2Seq Multi-Task Regularization (AMB+AE / FREQ+AE): Hierarchical LSTMs are expressive and prone to severe overfitting on small, noisy corpora. To fix this, the authors add an auxiliary character-level sequence-to-sequence autoencoder (reproducing an input word character-by-character) sharing the subword character-biLSTM encoder with the tagger.
Datasets
- Target Languages: 15 truly low-resource languages spanning 6 language families: Amharic, Belarusian, Breton, Buryat, Faroese, Upper Sorbian, Armenian, Kazakh, Kurmanji, Lithuanian, Marathi, Maltese, Tamil, Telugu, and Tagalog. Evaluation uses test splits from Universal Dependencies (UD) v2.1.
- High-Resource Comparison Set: German, Spanish, Italian, Portuguese, and Swedish evaluated on the PUD treebanks.
- External Resources: Unlabeled Wikipedia dumps, Polyglot word embeddings, bilingual wordlists from PanLex Swadesh and Wikidata, and morphological tags from UniMorph (available for only 3 of the 15 languages).
Key Insights and Ideas
- The "Low-Resource" Illusion: Prior successes in weakly supervised tagging were driven by dictionary coverage and dataset cleanliness rather than robust transfer algorithms. When tested on actual low-resource conditions, performance drops dramatically.
- Word-Level Auxiliary Tasks Hurt, Character Tasks Help: While language-modeling auxiliary losses (PLA16) help high-resource taggers, they hurt performance in low-resource environments because word frequencies cannot be estimated accurately from tiny corpora. Conversely, character autoencoding (AE) successfully regularizes character embeddings against noisy supervision.
- Severe Label Imbalance from Dictionaries: Naive dictionary projection produces skewed tag distributions. In FREQ, punctuation and verbs were dramatically over-represented compared to true test distributions, while nouns were heavily under-represented.
Key Results
- All models struggle: Across all 15 languages, no system achieved even 50% macro-average accuracy.
- Baseline performance: Unsupervised clustering (CHR11) averaged 34.40%, while the strongest existing baseline (GAR13) achieved 39.11%. PLA16 reached 38.99%.
- Proposed approaches: The single-task neural models scored 41.06% (FREQ) and 41.38% (AMB). Adding character-level autoencoding yielded the top overall performance at 42.25% (AMB+AE).
- High-Resource Sanity Check: When evaluated on high-resource European languages using standard, high-quality Wiktionary annotations ("Wiki-ly"), AMB jumped to 72% and FREQ reached 77% (compared to 43% and 50% when restricted to low-resource-style dictionaries).
Limitations and Future Work
- English Pivot Bias: The authors used English as the single source language for cross-lingual transfer due to dictionary availability, rather than choosing linguistically related, typological donor languages.
- Contextualized Embeddings Omitted: Pretrained language models (like mBERT or XLM-R) were excluded due to poor language identification, heavy spelling variation, and lack of pretraining data for genuine low-resource languages.
- Future Direction: The authors emphasize that minor architectural tweaks on top of existing supervised frameworks will not bridge this performance gap; truly low-resource POS tagging requires radically different paradigms for distant supervision and syntax induction.
Summary updated