Abstract
Part-of-speech (POS) taggers for low-resource languages which are exclusively based on various forms of weak supervision - e.g., cross-lingual transfer, type-level supervision, or a combination thereof - have been reported to perform almost as well as supervised ones. However, weakly supervised POS taggers are commonly only evaluated on languages that are very different from truly low-resource languages, and the taggers use sources of information, like high-coverage and almost error-free dictionaries, which are likely not available for resource-poor languages. We train and evaluate state-of-the-art weakly supervised POS taggers for a typologically diverse set of 15 truly low-resource languages. On these languages, given a realistic amount of resources, even our best model gets only less than half of the words right. Our results highlight the need for new and different approaches to POS tagging for truly low-resource languages.
Introduction
Part-of-Speech (POS) tagging is a fundamental building block for higher-level NLP tasks like machine translation. While supervised models for major languages like English achieve near-perfect accuracy (97%+), most of the world's 7,000+ languages lack the massive, manually annotated datasets required for such supervision. To bridge this gap, researchers have turned to "weakly supervised" methods that rely on cross-lingual transfer (projecting tags from a rich language like English) or tag dictionaries.
However, a significant gap exists between simulated and real-world low-resource NLP. Much of the existing research is conducted on "mid-resource" languages (e.g., Spanish or German) where data is artificially restricted to simulate scarcity. These languages still benefit from high-quality dictionaries and clean Wikipedia text. This study investigates how these models perform on truly low-resource languages like Amharic, Buryat, or Upper Sorbian, where resources are genuinely scarce, noisy, and typologically distant from English.
Methods
The study evaluates several baseline models and proposes an improved multi-task learning architecture:
- Cross-Lingual Transfer: Bilingual dictionaries from PanLex and Wikidata are used to project POS tags from English to the target language. Two strategies are tested: FREQ (tagging a word with the most frequent POS tag of its English translations) and AMB (tagging a word with all possible POS tags of its translations, treating the label as ambiguous).
- Hierarchical LSTMs: A state-of-the-art Bi-LSTM architecture processes both character and word levels.
- Multi-Task Learning (MTL) with Autoencoding: To prevent overfitting on sparse "silver-standard" data, an auxiliary task of word autoencoding (reconstructing input words character-by-character) is added. This forces the character-level encoder to learn more robust representations of the language's morphology.
Datasets
The study uses a diverse set of 15 truly low-resource languages representing six different language families (e.g., Afro-Asiatic, Dravidian, Turkic). Evaluation is performed on Universal Dependencies (UD) treebanks (v2.1), many of which had zero training data available.
As shown in Figure 1, the resource availability for these languages is significantly lower than for the high-resource control group, particularly regarding UNIMORPH entries and Wikipedia sentence counts, which are often used as proxies for "low-resource" settings in other literature.
Results
The experimental results provide a sobering reality check for weakly supervised methods. The best-performing model (AMB + Autoencoding) achieved a macro-average accuracy of only 42.25% across the 15 target languages.
Key insights from the results in Figure 3 include:
- The Dictionary Bottleneck: In real-world scenarios, dictionaries are sparse and noisy. The "ambiguous" (AMB) approach proved more robust than the frequency-based (FREQ) approach because it avoids committing to a single, potentially incorrect tag during training.
- The Importance of Characters: Character-level LSTMs are essential for capturing morphological signals (prefixes/suffixes) in low-resource settings where word-level data is missing (the out-of-vocabulary problem).
- Regularization via Autoencoding: The MTL approach with word reconstruction acts as a powerful regularizer, helping the model generalize better to unseen words without requiring additional labeled data.
- The "Low-Resource" Illusion: When the same setup was applied to high-resource languages with high-quality dictionaries, accuracy jumped to 77%. This confirms that the primary challenge is not just the model architecture, but the quality and quantity of underlying linguistic resources.
Conclusion
The performance of state-of-the-art weakly supervised POS taggers on truly low-resource languages is significantly lower than previously suggested by studies on simulated low-resource settings. Achieving less than 50% accuracy highlights a massive gap in current NLP capabilities. The authors call for "radically different approaches," potentially involving better cross-lingual word embeddings or leveraging native speaker expertise in active learning loops, rather than relying on the accidental availability of Wikipedia or PanLex data.
Summary updated