TL;DR
This paper explores how semi-supervised part-of-speech (POS) tagging, which requires less annotated data than traditional methods, performs in real-world speech applications. They found that it's a viable alternative to supervised tagging, especially in new domains, and can even offer gains when there's a mismatch between training and application data.
What Problem Does This Paper Address?
Part-of-speech (POS) tagging, the process of assigning grammatical categories (like noun, verb, adjective) to words, is crucial for many natural language processing tasks, including named entity recognition, semantic parsing, and speech recognition post-processing. However, high-performance POS taggers typically rely on large amounts of manually annotated data, which is expensive and time-consuming to create, especially for new domains or languages. This paper addresses the challenge of achieving good POS tagging performance with limited or no annotated data by investigating semi-supervised methods and their impact on downstream speech applications.
Methods
The authors propose a semi-supervised POS tagging approach based on Bayesian estimation for training a Hidden Markov Model (HMM). This method leverages a dictionary of possible tags for each word and learns from unannotated text. A key challenge in semi-supervised learning is handling out-of-vocabulary (OOV) words. To address this, they employ a supervised method using an AdaBoost classifier to predict possible tags for new words based on their morphological features (like prefixes and suffixes). They then evaluate this semi-supervised tagger on three downstream tasks: named entity tagging, semantic role labeling, and ASR output post-processing. They also investigate the impact of reducing the size of the initial word-tag dictionary.
Datasets
- CoNLL 2000 POS tagging corpus: Used for initial sanity checks of the semi-supervised tagger. It contains 221k words for training and 47k for testing, with 44 tags.
- ESTER corpus: A large corpus of French broadcast news transcripts (1.1M words) used for more realistic evaluation. Two small sets (test1 and test2) of 1000 words each were manually annotated with POS tags.
- MEDIA corpus: Used for semantic role labeling experiments, containing French telephone conversations annotated with flat concepts.
- LIUM system's ASR output: Used for the ASR output post-processing task, generated from the ESTER dataset.
Key Insights and Ideas
- Semi-supervised tagging is a strong contender: The paper demonstrates that semi-supervised POS tagging can achieve performance comparable to, and sometimes even better than, supervised taggers, especially when dealing with domain mismatch.
- Morphological features are key for OOV words: Predicting tags for unseen words using morphological features is crucial for maintaining performance.
- Downstream tasks absorb tagger errors differently: Tasks that heavily rely on the exact meaning of POS tags (like ASR post-processing) are more sensitive to tagger errors than tasks where the downstream system can learn from tagger mistakes (like named entity tagging).
- Dictionary size matters, but less than expected: While a smaller dictionary leads to performance degradation, the impact is mitigated by the semi-supervised approach and morphological prediction, especially for tasks that can tolerate some errors.
Key Results
- On the CoNLL 2000 dataset, the semi-supervised tagger with predicted ambiguity classes achieved an accuracy of 93.6%, close to the supervised tagger's 95.1%.
- For French broadcast news (ESTER), the supervised tagger achieved accuracies of 91.7% (test1) and 87.3% (test2), while the semi-supervised tagger with a full dictionary reached 80.8% and 77.1% respectively. However, when the dictionary was reduced to 5k or 2k words, performance dropped significantly.
- In Named Entity Tagging, semi-supervised POS tags often yielded comparable or better F-scores than supervised tags, particularly when using ASR output, likely due to the downstream system's ability to learn tagger errors.
- For Semantic Role Labeling, POS tags provided less of a boost compared to named entity tagging, as the task was more lexically driven. Semi-supervised and supervised tags showed similar gains.
- In ASR output post-processing, the semi-supervised tagger performed nearly as well as the supervised tagger, even with drastically reduced dictionaries, showing a nearly 21.5% relative reduction in word error rate.
Limitations and Future Work
- Dictionary size impact: While the semi-supervised approach mitigates the impact of small dictionaries, performance still degrades, especially for tasks that require high POS tag accuracy.
- ASR lattice integration: The authors suggest that a promising future direction is to adapt semi-supervised tagging to efficiently handle ASR lattices, which represent multiple possible word sequences from the speech recognition process.
Summary updated