Introduction
In Natural Language Processing (NLP), a "lexicon" serves as a dictionary of known words. However, no lexicon is ever truly complete—new words, names, and technical jargon appear constantly. When a system encounters an "unknown word," it must guess its Part-of-Speech (POS) to maintain the grammatical integrity of the sentence. This paper introduces a statistical classifier that leverages a loglinear model to integrate multiple linguistic features effectively, overcoming the limitations of previous independence-based models.
The Problem of Feature Independence
A significant challenge in early POS tagging models was the assumption of feature independence. Previous models treated word features—such as capitalization or specific suffixes—as independent variables. While this simplifies computation, linguistic features are often highly correlated.
When independence is assumed, adding more features can paradoxically degrade performance. The model may "over-count" evidence or become confused by redundant information. As illustrated in Figure 1, while the accuracy of a simple probabilistic model peaks and then declines as more features are added, the proposed statistical classifier maintains stability and continues to improve.
Methodology: Loglinear Smoothing
Alexander Franz proposes a classifier that moves beyond the independence assumption using three core techniques:
- Contingency Tables: The system counts how often specific features (like suffixes) co-occur with specific POS tags in a training corpus.
- Loglinear Smoothing: To address the "sparse data" problem—where many feature combinations never appear in training—a loglinear model is used to smooth these counts. This allows the model to estimate the probability of a tag even when the specific feature combination is rare.
- Conditional Probability: The classifier aims for a "minimum error rate" by maximizing the conditional probability of a classification given the observed features, following classic pattern classification theory.
Key Results and Discussion
The evaluation compared a baseline (prior probabilities), a reimplementation of the independent feature model, and the new statistical classifier using both 4 and 9 features. The results, summarized in Figure 3, demonstrate a clear progression in performance.
The statistical classifier achieved a substantial jump to 73% overall accuracy with 9 features, compared to just 61% for the 4-feature independent model. Notably, when the model was allowed to provide its "2-best" guesses, accuracy reached 87%. This high performance in narrowing down possibilities is particularly valuable for downstream parsing tasks where absolute certainty isn't always required in the first pass.
Conclusion
The primary insight of this work is the robustness of the loglinear model to "nuisance features." By accounting for feature interactions, researchers can utilize a wider array of linguistic clues without risking model collapse. This approach not only improves accuracy for unknown words but also suggests a path forward for other ambiguity resolution problems in NLP, such as Prepositional Phrase (PP) attachment.
Summary updated