Abstract
This paper investigates several augmented mixture models that are competitive alternatives to standard Bayesian models and prove to be very suitable to word sense disambiguation and related classification tasks. We present a new classification correction technique that successfully addresses the problem of under-estimation of infrequent classes in the training data. We show that the mixture models are boosting-friendly and that both Adaboost and our original correction technique can improve the results of the raw model significantly, achieving state-of-the-art performance on several standard test sets in four languages. With substantially different output to Naive Bayes and other statistical methods, the investigated models are also shown to be effective participants in classifier combination.
Introduction
Lexical disambiguation—the task of selecting the correct meaning or spelling of a word based on its context—remains a fundamental challenge in Natural Language Processing. Whether distinguishing between the legal and social senses of "bar" (Word Sense Disambiguation, WSD) or correcting a "peace/piece" typo (Context-Sensitive Spelling Correction, CSSC), models must navigate high-dimensional feature spaces and sparse data.
While Naive Bayes is a common baseline, it suffers from two primary issues: the "zero-frequency" problem where unseen feature combinations zero out probabilities, and a bias toward frequent classes that often ignores rare but correct senses. This research introduces Augmented Mixture Models as a robust alternative, combining a mixture model framework with a novel error-correction technique called Maximum Variance Correction (MVC) and a specialized boosting approach (AdaMixt).
Methodology
Unlike Naive Bayes, which multiplies feature probabilities, Mixture Models (MM) treat the conditional probability of a sense as a weighted sum (mixture) of individual feature probabilities. This additive nature makes them inherently more robust to missing features.
Maximum Variance Correction (MVC)
A core contribution of this work is MVC, a post-processing step designed to promote infrequent classes. Standard models often under-estimate rare senses. MVC shifts the focus from the absolute highest probability to "statistical surprise." By measuring the deviation of a sense's probability in a specific context from its typical behavior in the training set using variational coefficients, the model can identify when a rare sense is significantly more likely than expected.
Feature Engineering
The models utilize a rich feature set including context features (POS tags and lemmas), local bigrams, and syntactic predicate-argument relationships (e.g., identifying the object of a specific verb).
Evaluation and Results
The models were evaluated on SENSEVAL-1 and SENSEVAL-2 benchmarks for WSD across four languages (English, Swedish, Spanish, and Basque), and the Brown Corpus for CSSC.
Performance Gains
The inclusion of MVC consistently improved classification accuracy across all languages. On the SENSEVAL-2 English dataset, the MMVC model achieved 66.72% accuracy, outperforming feature-enhanced Naive Bayes (65.70%).
In the spelling correction task, the model reached 92.2% accuracy on 14 common confusion sets. Furthermore, the unique error profiles of Mixture Models made them excellent candidates for ensemble methods. Adding MMVC to a classifier combination reduced error rates by 3.5% to 4.5%.
Discussion and Conclusion
This research demonstrates that WSD and CSSC can be treated as the same fundamental machine learning problem. Mixture models provide a more graceful handling of sparse data than multiplicative models. The core insight of MVC lies in recognizing that a sense is often correct not because it is the most likely overall, but because its presence is contextually surprising.
While MVC requires sufficient training data to establish reliable variance statistics, its language-independent success suggests broad applicability. Future work will explore applying these augmented mixtures to other high-dimensionality tasks such as named-entity recognition and machine translation.
Summary updated