Abstract
Web-scale data has been used in a diverse range of language research. Most of this research has used web counts for only short, fixed spans of context. We present a unified view of using web counts for lexical disambiguation. Unlike previous approaches, our supervised and unsupervised systems combine information from multiple and overlapping segments of context. On the tasks of preposition selection and context-sensitive spelling correction, the supervised system reduces disambiguation error by 20-24% over the current state-of-the-art.
Introduction
Lexical disambiguation—the task of selecting the correct word for a specific context—is a fundamental challenge in Natural Language Processing (NLP). Whether correcting a typo like "affect" vs. "effect" or choosing the appropriate preposition in a sentence, the problem traditionally requires a trade-off between high-quality, linguistically-rich small corpora and the massive but noisy scale of the web. While search engine "hit counts" have been used in the past, they are often limited by query quotas and restricted to fixed-length context windows. This paper introduces a unified approach that leverages a pre-computed web-scale N-gram corpus to utilize multiple, overlapping context windows simultaneously.
Methodology
The core of this research involves two models utilizing the Google Web 5-gram Corpus, which contains counts for sequences of one to five words across the public web.
SUPERLM (Supervised)
This model employs a linear Support Vector Machine (SVM) to assign weights to 14 different "context patterns." These patterns consist of overlapping 2, 3, 4, and 5-grams that span the target word. By using aggregate counts as features rather than specific word features, the model has very few parameters, allowing it to reach peak performance with significantly less training data than traditional lexical models.
SUMLM (Unsupervised)
SUMLM is a simpler approach that sums the log-counts of all overlapping N-grams for each candidate word. Despite its simplicity, this unsupervised method performs remarkably well, demonstrating that at web-scale, data volume can often compensate for algorithmic complexity.
The Power of Overlapping Context
A key innovation of this work is the use of multi-scale context. Previous methods often relied on a single trigram. By shifting windows and varying N-gram lengths, the models capture more nuanced syntactic and semantic information.
As shown in the table above, increasing the maximum N-gram length from 2 to 5 significantly boosts accuracy, particularly when combining multiple lengths (e.g., Min 3, Max 5).
Experimental Results
Preposition Selection
On the task of preposition selection, SUPERLM achieved an accuracy of 75.4%. This is a significant milestone as it matches the measured human agreement rate of 75% on the same dataset.
Context-Sensitive Spelling Correction
Evaluating difficult confusion sets (e.g., raise vs. rise), the models achieved an average accuracy of 95.7%. This represents a 24% relative reduction in error compared to previous state-of-the-art unsupervised methods.
Pronoun Detection
The system was also applied to non-referential pronoun detection (identifying when "it" is a placeholder vs. a referent). It reached 82.4% accuracy, outperforming previous specialized models.
Conclusion
This research demonstrates that a unified view of lexical disambiguation, powered by web-scale N-gram data and overlapping context windows, can match or exceed human-level performance on complex linguistic tasks. The efficiency of these models—requiring fewer training examples and providing faster results than search engine APIs—makes them highly practical for a wide range of NLP applications. Future work may extend these techniques to dynamic candidate sets, such as those found in anaphora resolution.
Summary updated