Overcoming the Brittleness Bottleneck using Wikipedia: Enhancing Text Categorization with Encyclopedic Knowledge

Evgeniy Gabrilovich, Shaul MarkovitchAAAI 2006

Introduction

Traditional text categorization (TC) systems predominantly rely on the Bag of Words (BOW) model, which represents documents as simple frequency counts of their constituent terms. While computationally efficient, BOW models are fundamentally "brittle" because they lack external context. For instance, a model might fail to link "Bernanke" to the "Federal Reserve" or "Ciprofloxacin" to "antibiotics" unless these specific associations are explicitly present in the training data. This lack of world knowledge hinders performance on short documents and tasks involving synonymy or polysemy.

This research introduces a method to augment text classifiers with encyclopedic knowledge derived from Wikipedia. By mapping text fragments to a high-dimensional concept space, the proposed approach creates a "knowledge-rich" feature set that provides the context necessary for more human-like categorization.

Methodology: Wikipedia as a Concept Space

The core of the proposed system is a feature generation engine that utilizes Wikipedia as a structured proxy for human knowledge. The authors chose Wikipedia over other structured sources like the Open Directory Project (DMOZ) because Wikipedia articles are written in standard, "clean" English and possess a dense link structure that provides a richer web of conceptual relationships. The methodology involves several distinct phases:

  1. Concept Definition: Every Wikipedia article is treated as a distinct "concept." Using a 2005 snapshot, the authors curated a set of approximately 171,000 high-quality concepts by filtering out stubs and date-specific pages.
  2. Feature Generation: An auxiliary centroid classifier acts as a mapping function. Given a text segment, it identifies the top 10 most relevant Wikipedia concepts.
  3. Multi-Resolution Analysis: To address word sense disambiguation, the system analyzes text at multiple granularities—words, sentences, and paragraphs. For example, the term "Jaguar" is disambiguated by its proximity to words like "engine" (car) versus "jungle" (animal).
  4. Augmentation and Selection: The titles of relevant Wikipedia articles are appended as new features to the original BOW representation. Standard feature selection techniques are then applied to maintain a discriminative and manageable feature set.

Experimental Setup

The method was evaluated against five benchmark datasets: Reuters-21578, RCV1, OHSUMED (medical abstracts), 20 Newsgroups (20NG), and Movie Reviews. To test the system's robustness in data-sparse scenarios, the researchers also created "short document" variants of these datasets using only the document titles.

Results and Discussion

The Wikipedia-augmented approach consistently outperformed the BOW baseline across all datasets, demonstrating that providing a "cornucopia of world knowledge" can help machines compensate for limited reasoning skills.

Performance on Full Documents

As shown in Table 1, the integration of Wikipedia concepts led to substantial gains in both micro- and macro-averaged Break-Even Points (BEP). Notably, the macro-averaged performance improved by up to 30.4% on RCV1 and 18.0% on OHSUMED. These results suggest that external knowledge is particularly effective for categories with fewer training examples, where the model can leverage Wikipedia to bridge gaps in the training data.

Table 1: Performance comparison on full-text documents. The Wikipedia-augmented model shows significant improvements over the baseline, particularly in macro-averaged scores.

The Short Document Advantage

The impact of encyclopedic knowledge was most pronounced in short documents. In scenarios where the BOW model has minimal text to analyze, Wikipedia features provide a critical context boost. As seen in Table 2, for certain OHSUMED categories, performance improved by over 35% when using only titles.

Table 2: Performance comparison on short documents (titles only). The relative improvement is even more dramatic here, demonstrating the power of external knowledge in data-sparse environments.

Conclusion

By augmenting traditional classifiers with Wikipedia concepts, this research effectively addresses the brittleness of BOW models. While the system introduces computational complexity due to the scale of the Wikipedia feature space, its ability to resolve ambiguity and handle short texts marks a significant step forward. Future work will explore deeper link analysis within Wikipedia and the application of this knowledge-rich representation to broader natural language processing tasks.

Summary updated

BibTeX
@inproceedings{gabrilovich2006overcoming,
  author = {Evgeniy Gabrilovich and Shaul Markovitch},
  title = {{Overcoming the Brittleness Bottleneck using Wikipedia: Enhancing Text Categorization with Encyclopedic Knowledge}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure