Abstract This paper discusses the use of unlabeled examples for the problem of named entity classification. A large number of rules is needed for coverage of the domain, suggesting that a fairly large number of labeled examples should be required to train a classifier. However, we show that the use of unlabeled data can reduce the requirements for supervision to just 7 simple "seed" rules. The approach gains leverage from natural redundancy in the data: for many named-entity instances both the spelling of the name and the context in which it appears are sufficient to determine its type. We present two algorithms. The first method uses a similar algorithm to that of (Yarowsky 95), with modifications motivated by (Blum and Mitchell 98). The second algorithm extends ideas from boosting algorithms, designed for supervised learning tasks, to the framework suggested by (Blum and Mitchell 98).
Introduction
In the late 1990s, the dominant paradigm for Natural Language Processing (NLP) was supervised learning, which necessitated massive amounts of hand-labeled data. For Named Entity Classification (NEC), a model must analyze a string like "Honduras" or "Mrs. Frank" and categorize it as a Location, Person, or Organization.
The challenge lies in the vastness of language; covering every possible name and context theoretically requires thousands of labeled examples. Michael Collins and Yoram Singer addressed this "supervision bottleneck" by investigating whether massive amounts of unlabeled text could be used to train a model automatically. Their work demonstrates that NEC can be achieved with over 91% accuracy using almost no labeled data by leveraging the principle of co-training.
Methodology
The authors propose two primary unsupervised algorithms based on the principle of redundancy. In many sentences, both the spelling (e.g., "Mr. Cooper") and the context (e.g., "...says X, a vice president...") are independently sufficient to identify the entity type.
- DL-CoTrain (Decision List Co-Training): This is a "cautious" iterative approach. It starts with a small set of seed rules and uses them to label a portion of the unlabeled data. It then identifies the most reliable contextual rules from those new labels, which in turn are used to find more spelling rules. By adding only a few high-precision rules at each step, the model avoids "drifting" into error.
- CoBoost: This algorithm adapts AdaBoost to the unsupervised setting. It treats the disagreement between the spelling classifier and the context classifier as a cost to be minimized, essentially forcing two different "views" of the data to agree with each other on unlabeled examples.
- EM Baseline: These methods were compared against a standard Expectation-Maximization (EM) approach using a generative Naive Bayes model.
Experimental Setup
The training data consisted of 971,746 sentences from the New York Times, processed to extract roughly 90,000 (spelling, context) pairs. Crucially, the models were initialized with only 7 simple seed rules, as shown below:
Evaluation was performed on a set of 1,000 randomly selected examples, hand-labeled by the authors. After removing temporal expressions, 962 examples remained for testing.
Key Results and Insights
The results represented a breakthrough for unsupervised NLP. The DL-CoTrain algorithm achieved 91.3% accuracy on clean data, while CoBoost followed closely at 91.1%. Both significantly outperformed the standard EM approach (83.1%) and the original Yarowsky algorithm (81.3%). For comparison, a baseline of simply guessing "Organization" for every entity yielded only 45.8% accuracy.
The core insight is the "Power of Two Views." Names have a dual nature: "Mr. Cooper" identifies a person regardless of context, while "the president of X" identifies X as an organization regardless of its name. If a model sees "Mr. Cooper, the president of S.&P.," it can use the "Mr." to learn that "president of..." is a person-context. Furthermore, the authors found that "cautiousness" is vital; limiting rule growth per iteration (as in DL-CoTrain) provides much higher stability than aggressive rule addition.
Limitations and Future Work
Despite the high accuracy, the models struggled with "noise"—items that did not fit into the three main categories (e.g., "The 1990s"). When noise was included, accuracy dropped by approximately 8-10%. Additionally, this research assumes names have already been segmented from the text. Future work should aim to combine segmentation and classification into a single unsupervised system. Finally, the poor performance of the EM model highlighted the pitfalls of assuming feature independence in linguistic data, as it failed to recognize that words like "New" and "York" are highly correlated.
Summary updated