Limitations of Co-Training for Natural Language Learning from Large Datasets

David R. Pierce, Claire CardieEMNLP 2001 · OpenReview

Abstract

Co-Training is a weakly supervised learning paradigm in which the redundancy of the learning task is captured by training two classifiers using separate views of the same data. This enables bootstrapping from a small set of labeled training data via a large set of unlabeled data. This study examines the learning behavior of co-training on natural language processing tasks that typically require large numbers of training instances to achieve usable performance levels. Using base noun phrase bracketing as a case study, we find that co-training reduces by 36% the difference in error between co-trained classifiers and fully supervised classifiers trained on a labeled version of all available data. However, degradation in the quality of the bootstrapped data arises as an obstacle to further improvement. To address this, we propose a moderately supervised variant of cotraining in which a human corrects the mistakes made during automatic labeling. Our analysis suggests that corrected co-training and similar moderately supervised methods may help cotraining scale to large natural language learning tasks.

Introduction

Co-training is a powerful "weakly supervised" learning algorithm designed to leverage vast amounts of unlabeled data. By assuming two independent "views" of the same data—such as the text on a webpage versus the anchor text of links pointing to it—two classifiers can be trained to "teach" each other. While this approach has proven successful for relatively simple classification tasks, its scalability to complex natural language processing (NLP) problems remains a significant question. Many core NLP tasks, such as identifying noun phrases, require hundreds of thousands of training examples to reach production-level accuracy. This study investigates whether co-training can bridge this gap or if it eventually succumbs to the noise generated during the bootstrapping process.

Methodology and Experimental Setup

The study focuses on Base Noun Phrase (NP) Identification using the Penn Treebank Wall Street Journal corpus. The task is decomposed into two distinct views:

  1. Left Context: Features derived from words and parts-of-speech to the left of a target word.
  2. Right Context: Features derived from words and parts-of-speech to the right.

Naive Bayes classifiers are employed for each view. To adapt standard co-training for the complexities of NLP, the authors introduced a modified selection process that maintains the natural distribution of labels (IOB tagging: Inside, Beginning, or Outside a noun phrase). This prevents the model from focusing solely on "easy" examples and ignoring rare but critical cases.

The training set consisted of approximately 211,000 instances, with a separate test set of 47,000 instances. The ultimate goal was to match the performance of a fully supervised model, which achieved an accuracy of 95.17%.

The Performance Peak and Data Degradation

Experimental results reveal that co-training performance is not monotonic. Accuracy typically improves for a limited period—roughly 160 iterations in this case study—before beginning a steady decline. During this initial phase, co-training successfully reduced the error gap between the baseline and the fully supervised goal by 36%, reaching a peak accuracy of roughly 92.5%.

However, the primary obstacle to further improvement is the degradation of data quality. As the two classifiers exchange labels, they inevitably begin to "hallucinate" or feed each other incorrect examples. This creates a feedback loop of noise that eventually outweighs the benefits of the additional data.

Figure 3: (a) Classifier accuracy peaks and then declines as (b) the quality of newly labeled data degrades over iterations.

Sensitivity to Initial Labeled Data

The success of co-training is highly sensitive to the size of the initial labeled dataset ($L$). This presents a "Goldilocks" problem:

  • If $L$ is too small (e.g., 200), the initial classifiers are too weak to provide reliable labels for the unlabeled pool, leading to rapid degradation.
  • If $L$ is too large (e.g., 1000), the classifiers are already sufficiently accurate that the additional information gained through co-training is marginal.
Figure 5: Comparison of co-training performance with (a) L=200 and (b) L=1000 initial labeled instances.

There is a fundamental tension between confidence and coverage. To maintain high data quality, the algorithm should select examples the classifier is most confident about. However, these are often examples the model already "knows." To learn new patterns, the model must encounter unfamiliar examples, which it is more likely to label incorrectly.

Corrected Co-Training: A Moderately Supervised Solution

To overcome the noise ceiling, the authors proposed Corrected Co-Training. In this "moderately supervised" variant, a human annotator reviews and corrects the labels generated by the classifiers before they are added to the training set.

This approach proved remarkably effective, reaching an accuracy of 95.03%, nearly matching the fully supervised ceiling. Crucially, this was achieved with significantly less human effort: the annotator only had to correct approximately 450 mistakes out of 8,000 reviewed instances. This represents a massive efficiency gain over labeling the entire 211,000-instance corpus from scratch.

Conclusion and Future Work

While standard co-training provides a useful boost for NLP tasks, its utility is limited by the eventual accumulation of labeling errors. The study demonstrates that human-in-the-loop interventions, such as Corrected Co-Training, are essential for scaling these methods to large-scale, complex datasets. Future research may explore combining co-training with Active Learning, where human effort is focused specifically on the most "uncertain" or informative examples rather than just the most "confident" ones, further optimizing the balance between supervision and automation.

Summary updated

Abstract

Co-Training is a weakly supervised learning paradigm in which the redundancy of the learning task is captured by training two classifiers using separate views of the same data. This enables bootstrapping from a small set of labeled training data via a large set of unlabeled data. This study examines the learning behavior of co-training on natural language processing tasks that typically require large numbers of training instances to achieve usable performance levels. Using base noun phrase bracketing as a case study, we find that co-training reduces by 36% the difference in error between co-trained classifiers and fully supervised classifiers trained on a labeled version of all available data. However, degradation in the quality of the bootstrapped data arises as an obstacle to further improvement. To address this, we propose a moderately supervised variant of cotraining in which a human corrects the mistakes made during automatic labeling. Our analysis suggests that corrected co-training and similar moderately supervised methods may help cotraining scale to large natural language learning tasks.
BibTeX
@inproceedings{pierce2001limitations,
  author = {David R. Pierce and Claire Cardie},
  title = {{Limitations of Co-Training for Natural Language Learning from Large Datasets}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2001},
  url = {https://openreview.net/forum?id=By40nZz_-B},
}
Figure