TL;DR
This paper introduces a novel approach to Chinese Part-of-Speech (POS) tagging by enhancing a standard bigram Hidden Markov Model (HMM) with "latent annotations" and then applying self-training. This combination significantly boosts accuracy, achieving state-of-the-art results on the Penn Chinese Treebank, even outperforming more complex trigram models.
What Problem Does This Paper Address?
Part-of-Speech (POS) tagging, the task of assigning grammatical tags (like noun, verb, adjective) to words in a sentence, is a fundamental step in many Natural Language Processing (NLP) tasks. While English POS tagging has seen impressive accuracy (over 97%), Chinese POS tagging has been more challenging. Traditional HMM taggers often rely on strong independence assumptions (e.g., a word's tag only depends on the previous tag). While using more complex models like trigram HMMs or incorporating sophisticated smoothing can help, they often struggle with data sparsity. This paper aims to improve Chinese POS tagging accuracy by exploring a simpler bigram HMM framework, making it more robust and accurate.
Methods
The core innovation lies in two main areas:
- Latent Annotations: Instead of directly using observed POS tags, the authors introduce "latent annotations." This means they allow a single observed POS tag (e.g., "NR" for proper noun) to be split into finer-grained, unobserved (latent) sub-tags (e.g., "NR-1", "NR-2"). This allows the model to learn more nuanced distinctions within categories, similar to how latent variables are used in Probabilistic Context-Free Grammars (PCFGs). The model parameters (transition and emission probabilities between these latent tags) are learned using the Expectation-Maximization (EM) algorithm. A hierarchical split-and-merge strategy is employed to determine the optimal number and placement of these latent annotations, prioritizing splits that yield the greatest improvement in training data likelihood. They also introduce a special handling for rare words by mapping them to an "unk" token and redistributing their statistics.
- Self-Training: After training the bigram HMM with latent annotations on labeled data, the model is further improved using self-training. This involves using the trained tagger to label a large amount of unlabeled data. This newly labeled data is then combined with the original labeled data to retrain the tagger. This semi-supervised approach leverages abundant unlabeled text to enhance the model's performance.
Datasets
- Penn Chinese Treebank 6.0 (CTB6): This is the primary dataset used for training and evaluation. It's a standard benchmark for Chinese NLP tasks.
- Unlabeled Data: A large corpus of unlabeled Chinese sentences, selected from similar sources as the CTB6 (news articles, broadcast news transcriptions), was used for the self-training experiments. These sentences were normalized and word-segmented.
Key Insights and Ideas
- Latent Annotations Boost Simplicity: Surprisingly, a bigram HMM with latent annotations can outperform a more complex trigram HMM. This suggests that carefully learned latent structures can capture dependencies more effectively than simply increasing the order of the Markov model, especially when dealing with data sparsity.
- Self-Training Synergy: The combination of latent annotations and self-training is particularly powerful. While self-training alone offers limited benefits (or even hurts performance) for standard bigram or trigram taggers when sufficient labeled data is available, it significantly improves the latent-annotated bigram tagger.
- Targeted Latent Splits: The hierarchical split-and-merge approach for latent annotations is crucial. It intelligently identifies which POS tags (like "NN" and "VV") benefit most from finer granularity, leading to more efficient learning.
- Rare Word Handling is Key: Explicitly addressing rare words through a dedicated mechanism is important for maintaining accuracy, especially when dealing with large vocabularies and limited labeled data.
Key Results
- The bigram HMM with latent annotations (Bigram+LA) achieved a token accuracy of 94.53% on the test set.
- Further improving this model with self-training (Bigram+LA+ST) pushed the accuracy to 94.78%, which was the state-of-the-art result reported for Chinese POS tagging on the CTB6 at the time of publication.
- This performance significantly surpassed the baseline bigram tagger (92.25%) and the trigram tagger (93.99%).
- The improvements were statistically significant (p < 0.005).
Limitations and Future Work
- Initial Trigram Attempt: The authors mention that initially trying to add latent annotations to a trigram tagger did not work well. They hypothesize that this is because latent annotations exacerbate the sparsity problem for trigram word emissions, and a simple extension to trigram ignores whether additional context is truly supported by data.
- Future Work: The authors plan to investigate automatic data selection methods for self-training to choose materials that are most suitable and to evaluate the effect of the amount of automatically labeled data. They also suggest exploring how to integrate this approach with other advanced techniques.
Summary updated