Abstract
Most work on weakly-supervised learning for part-of-speech taggers has been based on unrealistic assumptions about the amount and quality of training data. For this paper, we attempt to create true low-resource scenarios by allowing a linguist just two hours to annotate data and evaluating on the languages Kinyarwanda and Malagasy. Given these severely limited amounts of either type supervision (tag dictionaries) or token supervision (labeled sentences), we are able to dramatically improve the learning of a hidden Markov model through our method of automatically generalizing the annotations, reducing noise, and inducing word-tag frequency information.
Introduction
Part-of-speech (POS) tagging is a foundational task in Natural Language Processing, yet most of the world's 6,000+ languages lack the massive gold-standard datasets required to train modern models. Previous research often simulated low-resource environments by subsampling large, clean datasets. However, real-world low-resource scenarios involve noisy, incomplete data created under severe time constraints.
This study investigates the most effective way to utilize a linguist's time when limited to exactly two hours. We explore whether it is better to label full sentences (token supervision) or provide a tag dictionary (type supervision) and propose a pipeline to maximize the utility of this sparse input.
Methodology
We propose a four-stage pipeline to bridge the gap between sparse human annotation and a robust tagger:
- Tag Dictionary Expansion: We use Label Propagation (LP) to spread human-provided tags across a 100k-word raw corpus. A graph is constructed where words (nodes) are connected based on shared contexts and morphological features like prefixes and suffixes.
- Weighted Model Minimization: To handle noise in the expanded dictionary, we employ a greedy algorithm to find the minimal set of tag-to-tag transitions (bigrams) that explain the raw text. These transitions are weighted by LP confidence scores to filter out improbable human or propagation errors.
- HMM Training: A Hidden Markov Model (HMM) is trained using Expectation-Maximization (EM), initialized by the noisy labels from the previous steps.
- MEMM Refinement: Finally, the HMM labels the raw corpus to create "silver" data for training a Maximum Entropy Markov Model (MEMM), which better handles sub-word features for unseen words.
Experimental Setup
We evaluated our approach on three languages:
- Kinyarwanda (KIN): A morphologically rich Niger-Congo language.
- Malagasy (MLG): An Austronesian language.
- English (ENG): A control language using the Penn Treebank.
In each case, the training data was restricted to what a linguist could produce in a strict 120-minute window.
Results and Analysis
The system achieved 71–78% accuracy across all languages using only two hours of effort. A key finding is that type supervision (dictionaries) consistently outperforms token supervision (sentences) in time-constrained settings. Dictionaries provide broader vocabulary coverage, whereas sentence labeling wastes time on repetitive common words.
As shown in the results table, the combination of Label Propagation and model minimization significantly boosts performance, particularly for unknown words. For example, in English, accuracy on unseen words jumped from a baseline of ~38% to 56% through our expansion and minimization techniques. The weighted minimization step acts as a powerful noise filter, automatically discarding improbable tags accidentally included by hurried human annotators.
Conclusion
This work demonstrates that high-quality POS taggers can be built for low-resource languages with minimal human effort. By prioritizing tag dictionaries over sentence labeling and using graph-based generalization, we can create effective models that survive on a "starvation diet" of data. Future work will explore incorporating existing noisy resources like Wiktionary into the expansion graph.
Summary updated