Abstract
Natural Language Processing (NLP) models frequently encounter the "data sparseness" problem, where many valid linguistic constructions are absent from even the largest standard corpora. This paper explores using the World Wide Web as a massive proxy corpus by leveraging search engine hit counts to estimate bigram frequencies. The results demonstrate that web-derived data correlates strongly with curated corpora like the British National Corpus (BNC) and significantly outperforms them in predicting human plausibility judgments for rare or "unseen" word combinations.
Introduction
In Natural Language Processing, models rely heavily on word frequencies to understand language patterns. However, even standard datasets like the 100-million-word British National Corpus (BNC) suffer from data sparseness. Many perfectly valid word combinations (bigrams) simply never appear in these datasets. If a model sees a count of zero for a phrase like "plasma nylon," it cannot distinguish between a phrase that is linguistically impossible and one that is merely rare. This "zero-count" problem limits the performance of algorithms in tasks like machine translation, word sense disambiguation, and parsing. This study investigates whether the sheer scale of the internet can be used to fill these gaps.
Methodology
The researchers proposed a simple yet effective heuristic: using search engines (specifically Altavista and Google) to retrieve frequency counts for three types of bigrams: adjective-noun, noun-noun, and verb-object.
- Querying: Exact-match queries were constructed for bigrams, accounting for basic inflections and common intervening words (e.g., "the" or "a" for verb-object pairs). For example, to check the verb-object pair fulfill obligation, they would query "fulfill obligation", "fulfill the obligation", and "fulfill an obligation".
- Validation: To ensure these counts were meaningful, the authors conducted a task-based evaluation comparing web-derived frequencies against human "plausibility judgments" collected via Magnitude Estimation (a psychophysical technique where subjects rate the "fit" of a word pair relative to a baseline).
Datasets and Evaluation
The study utilized the BNC as a "gold standard" baseline. At the time of research (circa 2002), the searchable web was estimated to contain between 33 and 97 billion words—roughly three orders of magnitude larger than the BNC. The evaluation set consisted of 540 bigrams: 90 "seen" and 90 "unseen" for each of the three syntactic categories.
Key Results
The primary finding was that scale trumps noise. While web data is unfiltered and linguistically unbalanced, its sheer volume more than compensates for the lack of cleanliness.
- High Correlation: Web frequencies correlated strongly with BNC frequencies ($r$ ranging from .675 to .849).
- Superior Plausibility Prediction: For "seen" bigrams, web counts were better predictors of human judgment than the BNC. For example, in noun-noun pairs, Google counts had a correlation of .693 with human ratings, compared to the BNC’s .517.
- Solving the Zero-Count Problem: For bigrams that appeared zero times in the BNC, web counts still showed a significant correlation with human judgments ($r$ between .446 and .611), providing a signal where traditional corpora provided none.
Discussion and Conclusion
The study demonstrates that the web is a viable and often superior source of frequency data for NLP tasks. The "unseen" is predictable: web counts can reliably predict human intuition for phrases that never appear in traditional corpora, suggesting that "zero" in a standard corpus is often a sampling artifact rather than a linguistic truth. Google, indexing a larger portion of the web at the time, proved slightly more robust than Altavista. Future work will explore more complex syntactic variations and compare web counts against sophisticated mathematical smoothing algorithms.
Summary updated