Simple Semi-Supervised Training of Part-Of-Speech Taggers

Anders SøgaardACL 2010 · OpenReview

Abstract

Anders Søgaard. Proceedings of the ACL 2010 Conference Short Papers.

2010.

Semi-supervised learning for Part-of-Speech (POS) tagging has historically struggled to consistently outperform supervised baselines due to the complexity of sequence models. This paper introduces a simplified approach that reframes sequence labeling as a classification task through "stacking." By combining this framework with a novel "tri-training with disagreement" algorithm, the author achieves a 4.21% error reduction on the Wall Street Journal corpus. Notably, the disagreement-based variant proves exceptionally efficient, requiring significantly less unlabeled data than standard tri-training.

Introduction

Part-of-Speech (POS) tagging is a fundamental task in Natural Language Processing. While supervised models trained on large corpora like the Wall Street Journal (WSJ) achieve high accuracy, the cost of manual annotation remains a barrier. Semi-supervised learning—leveraging small amounts of labeled data alongside large unlabeled datasets—offers a potential solution. However, complex sequence models like HMMs or CRFs often suffer from error propagation when trained on noisy, automatically labeled data, sometimes performing worse than their supervised counterparts. This work seeks a more robust, simpler method to harness unlabeled data without the negative results that have plagued the field for decades.

Methodology

The proposed system rests on two primary innovations:

1. Stacking: Tagging as Classification

To avoid the pitfalls of semi-supervised sequence modeling, the author employs "stacking." The process involves running a supervised tagger (SVMTool) and an unsupervised clustering algorithm (Unsupos) on the text. The outputs (predicted tags and cluster IDs) are then used as features for a second-stage meta-classifier. This transformation reduces the problem from sequence prediction to a standard classification task, which is more amenable to robust semi-supervised techniques.

2. Tri-training with Disagreement

Tri-training typically involves three classifiers ($c_1, c_2, c_3$). In standard tri-training, if $c_2$ and $c_3$ agree on a label for an unlabeled instance, that instance is added to the training set for $c_1$. The author introduces a "disagreement" constraint: an instance is only added to $c_1$’s training set if $c_2$ and $c_3$ agree and $c_1$ currently predicts a different label. To ensure the necessary diversity among classifiers, they are trained on different bootstrap samples of the labeled data. This approach focuses the learning process on instances where the model is likely to improve.

Results and Discussion

The effectiveness of these methods was evaluated using the WSJ portion of the Penn Treebank for labeled data and the Brown corpus for unlabeled data.

Table 4: Accuracy results for SVMTool and MaxEnt across various training regimes.

As shown in Table 4, the "tri-training with disagreement" (tri-disagr) method achieved the highest accuracy, resulting in a 4.21% error reduction over the SVMTool baseline. The system improved baseline accuracy from 97.15% to 97.27%. The approach also proved effective for a MaxEnt tagger, demonstrating its generalizability across different base learners.

Efficiency of Disagreement

A key finding of the study is the extreme efficiency of the disagreement-based selection compared to standard tri-training.

Table 3: Comparison of average tokens added to the training set in standard tri-training versus tri-training with disagreement.

While standard tri-training incorporated over 1.1 million tokens from the unlabeled corpus, the disagreement variant achieved comparable or better performance by adding only 173 tokens on average (Table 3). This suggests that the vast majority of unlabeled data is redundant in a semi-supervised context and that targeting model weaknesses is a far more efficient strategy.

Conclusion

This paper demonstrates that semi-supervised POS tagging can be made both simple and effective by treating it as a classification problem. The "tri-training with disagreement" method provides a significant boost in accuracy while being remarkably data-efficient. This "off-the-shelf" framework makes it easier to incorporate other types of unsupervised features, such as word embeddings or gazetteers, into the tagging process in the future.

Summary updated

Abstract

Anders Søgaard. Proceedings of the ACL 2010 Conference Short Papers. 2010.
BibTeX
@inproceedings{sgaard2010simple,
  author = {Anders Søgaard},
  title = {{Simple Semi-Supervised Training of Part-Of-Speech Taggers}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=UQQSwDExJta},
}
Figure