Augmented Mixture Models for Lexical Disambiguation

Silviu Cucerzan, David YarowskyEMNLP 2002 · OpenReview

Abstract

This paper investigates several augmented mixture models that are competitive alternatives to standard Bayesian models and prove to be very suitable to word sense disambiguation and related classification tasks. We present a new classification correction technique that successfully addresses the problem of under-estimation of infrequent classes in the training data. We show that the mixture models are boosting-friendly and that both Adaboost and our original correction technique can improve the results of the raw model significantly, achieving state-of-the-art performance on several standard test sets in four languages. With substantially different output to Naive Bayes and other statistical methods, the investigated models are also shown to be effective participants in classifier combination.

Introduction

Lexical disambiguation—the task of selecting the correct meaning or spelling of a word based on its context—remains a fundamental challenge in Natural Language Processing. Whether distinguishing between the legal and social senses of "bar" (Word Sense Disambiguation, WSD) or correcting a "peace/piece" typo (Context-Sensitive Spelling Correction, CSSC), models must navigate high-dimensional feature spaces and sparse data.

While Naive Bayes is a common baseline, it suffers from two primary issues: the "zero-frequency" problem where unseen feature combinations zero out probabilities, and a bias toward frequent classes that often ignores rare but correct senses. This research introduces Augmented Mixture Models as a robust alternative, combining a mixture model framework with a novel error-correction technique called Maximum Variance Correction (MVC) and a specialized boosting approach (AdaMixt).

Methodology

Unlike Naive Bayes, which multiplies feature probabilities, Mixture Models (MM) treat the conditional probability of a sense as a weighted sum (mixture) of individual feature probabilities. This additive nature makes them inherently more robust to missing features.

Maximum Variance Correction (MVC)

A core contribution of this work is MVC, a post-processing step designed to promote infrequent classes. Standard models often under-estimate rare senses. MVC shifts the focus from the absolute highest probability to "statistical surprise." By measuring the deviation of a sense's probability in a specific context from its typical behavior in the training set using variational coefficients, the model can identify when a rare sense is significantly more likely than expected.

Variational coefficients illustrating the Maximum Variance Correction (MVC) technique. The coefficients highlight senses that are statistically surprising in a given context.

Feature Engineering

The models utilize a rich feature set including context features (POS tags and lemmas), local bigrams, and syntactic predicate-argument relationships (e.g., identifying the object of a specific verb).

Evaluation and Results

The models were evaluated on SENSEVAL-1 and SENSEVAL-2 benchmarks for WSD across four languages (English, Swedish, Spanish, and Basque), and the Brown Corpus for CSSC.

Performance Gains

The inclusion of MVC consistently improved classification accuracy across all languages. On the SENSEVAL-2 English dataset, the MMVC model achieved 66.72% accuracy, outperforming feature-enhanced Naive Bayes (65.70%).

Sense classification accuracy across four languages, comparing the baseline (Most Likely) with Mixture Models (MM) and the augmented MMVC.

In the spelling correction task, the model reached 92.2% accuracy on 14 common confusion sets. Furthermore, the unique error profiles of Mixture Models made them excellent candidates for ensemble methods. Adding MMVC to a classifier combination reduced error rates by 3.5% to 4.5%.

Discussion and Conclusion

This research demonstrates that WSD and CSSC can be treated as the same fundamental machine learning problem. Mixture models provide a more graceful handling of sparse data than multiplicative models. The core insight of MVC lies in recognizing that a sense is often correct not because it is the most likely overall, but because its presence is contextually surprising.

While MVC requires sufficient training data to establish reliable variance statistics, its language-independent success suggests broad applicability. Future work will explore applying these augmented mixtures to other high-dimensionality tasks such as named-entity recognition and machine translation.

Summary updated

Abstract

This paper investigates several augmented mixture models that are competitive alternatives to standard Bayesian models and prove to be very suitable to word sense disambiguation and related classification tasks. We present a new classification correction technique that successfully addresses the problem of under-estimation of infrequent classes in the training data. We show that the mixture models are boosting-friendly and that both Adaboost and our original correction technique can improve the results of the raw model significantly, achieving state-of-the-art performance on several standard test sets in four languages. With substantially different output to Naive Bayes and other statistical methods, the investigated models are also shown to be effective participants in classifier combination.
BibTeX
@inproceedings{cucerzan2002augmented,
  author = {Silviu Cucerzan and David Yarowsky},
  title = {{Augmented Mixture Models for Lexical Disambiguation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2002},
  url = {https://openreview.net/forum?id=SyVQ5ffOZS},
}
Figure