An Overview of Microsoft Web N-gram Corpus and Applications

Kuansan Wang, Christopher Thrasher, Evelyne Viegas, Xiaolong Li, Bo-June Paul HsuNAACL 2010

Introduction

In the field of Natural Language Processing (NLP), the availability of large-scale datasets has historically been a primary driver of progress. While static datasets like the 2006 Google N-gram corpus provided a foundational resource, they suffer from two significant limitations: temporal obsolescence and stylistic homogeneity. As the web evolves, static corpora fail to capture emerging terminology, slang, and product names. Furthermore, treating all web text as a single "language" ignores the structural differences between document bodies, titles, and anchor text.

To address these challenges, Microsoft Research developed the Microsoft Web N-gram corpus. Unlike its predecessors, this corpus is delivered as a dynamic XML Web Service, allowing for continuous updates and providing distinct models for different web content streams.

Architecture and Methodology

The core innovation of the Microsoft Web N-gram corpus is its delivery as a cross-platform XML Web Service. This approach eliminates the need for researchers to download and host terabytes of data while ensuring they always have access to the most current models.

The service utilizes the Constantly Adaptive Language Modeling (CALM) algorithm. CALM allows the models to integrate new N-grams discovered by the Bing crawler in real-time, provided they meet specific statistical significance thresholds. This ensures the corpus remains a "living map" of the web. Additionally, the system employs smoothed back-off models rather than raw counts, providing an open-vocabulary resource that handles unseen tokens more gracefully than simple frequency-based approaches.

Corpus Composition

The corpus is derived from the EN-US market index of the Bing search engine. As of the June 2009 snapshot, the scale of the data is immense:

  • Document Body: ~1.4 trillion tokens
  • Anchor Text: ~357 billion tokens
  • Document Titles: ~12.5 billion tokens

The researchers argue that these three streams represent distinct "dialects" of the web. Anchor text and titles are often more concise and keyword-heavy, closely mirroring the way users formulate search queries. By separating these streams, the corpus enables more specialized NLP applications.

Applications

The utility of the multi-stream approach is demonstrated through two key tasks: query segmentation and word breaking.

Search Query Segmentation

Search query segmentation involves identifying the most likely boundaries between concepts in a user's query. For instance, in the query "mike siwek lawyer mi," a system must decide if "mike siwek" is a single entity or if "siwek lawyer" is the primary phrase.

Likelihood rankings for query segmentation across Title and Anchor text streams.

As shown in the figure above, the Title and Anchor models provide different log-probability distributions for various segmentations. In many search-related contexts, these specialized models yield more intuitive and statistically significant results than a model trained solely on standard document body text.

Word Breaking (The LongStringProblem)

Word breaking is the task of inserting spaces into concatenated strings, a common requirement when processing URLs or hashtags. This is often referred to as the "LongStringProblem."

Log-probability rankings for disambiguating concatenated strings.

The Microsoft Web N-gram corpus excels at disambiguating potentially problematic strings. For example, the string "expertsexchange" could be segmented as "experts exchange" or "expert sex change." Because the corpus reflects the statistical reality of professional web content, it correctly identifies "experts exchange" as the significantly more probable intended phrase. Similarly, it can decode SMS-style shorthand like "w84u" into "wait for you" by leveraging the raw, unstemmed nature of the web data.

Conclusion and Future Work

The Microsoft Web N-gram corpus represents a shift toward dynamic, context-aware language modeling. While highly effective, the authors note certain limitations, such as the tendency to leave common URL fragments unspaced if they appear frequently enough as single units in the body text.

Future research facilitated by this service will likely focus on "multi-style" language modeling, where systems learn to transition between the different linguistic styles of the web—from the formal prose of an article body to the shorthand of a hyperlink.

Summary updated

BibTeX
@inproceedings{wang2010overview,
  author = {Kuansan Wang and Christopher Thrasher and Evelyne Viegas and Xiaolong Li and Bo-June Paul Hsu},
  title = {{An Overview of Microsoft Web N-gram Corpus and Applications}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
}
Figure