Intelligence in Wikipedia

Daniel S. Weld, Fei Wu, Eytan Adar, Saleema Amershi, James Fogarty, Raphael Hoffmann, Kayur Patel, Michael SkinnerAAAI 2008

Introduction

While Wikipedia is a massive repository of human knowledge, much of it remains "trapped" in unstructured text. Infoboxes—the summary tables on the right side of articles—provide structured data, but they are often missing, incomplete, or inconsistent across different languages. The "Intelligence in Wikipedia" (IWP) project addresses this by creating a "virtuous cycle" that combines automated information extraction (IE) with human communal content creation (CCC).

The Virtuous Cycle

The core philosophy of IWP is that AI and humans should not work in isolation. AI bootstraps the content, humans provide the "gold standard" corrections, and those corrections are fed back into the AI to retrain and improve the models. By using AI to suggest infobox entries and inviting casual readers to verify them, the system builds a massive structured knowledge base while simultaneously generating high-quality training data to improve its own algorithms.

The virtuous cycle of information extraction and human edits.

Addressing the Long Tail

Standard machine learning requires many examples, but 82% of Wikipedia categories have fewer than 100 instances. This makes it difficult to train reliable extractors for niche topics like "Irish Newspapers." To solve this "long tail" problem, IWP introduces several interconnected systems:

  • Kylin (Self-Supervised IE): Kylin uses existing Wikipedia infoboxes as a "ground truth" to find corresponding sentences in article text, training classifiers to identify relevant sentences and extract specific attribute values.
  • KOG (Kylin Ontology Generator): KOG automatically builds a rich ontology by mapping Wikipedia infoboxes to WordNet, using Markov Logic Networks to infer relationships.
  • Shrinkage (Hierarchical Smoothing): This statistical technique allows sparse categories to "borrow" data from their parents in the ontology. For instance, if there is insufficient data for a "Newspaper" extractor, the system uses weighted data from the "Publication" parent class.

Mixed-Initiative Design

The researchers designed a low-friction interface where casual readers see a small icon or popup. This allows them to verify an AI-suggested fact with a single click, turning passive readers into "micro-contributors." The authors found that you don't need to recruit "power editors" for everything; by making the UI non-intrusive, they achieved one fact validation for every eight page visits.

A mixed-initiative UI allowing casual readers to verify extracted facts.

Key Results and Impact

The implementation of these systems yielded significant improvements in data extraction and verification:

  • Recall Boost: Using "Shrinkage" dramatically improved the system's ability to find information in sparse classes. Recall improved by 57% for "Performers" and 457% for "Irish Newspapers."
  • Web Augmentation: By looking outside Wikipedia to the general web for training data, the area under the Precision/Recall curve improved by over 1000% for some sparse categories.
  • Human Verification: The simple icon UI design led to voluntary contributions with 90% precision, proving that casual users can effectively "clean" AI-generated data.

Limitations and Future Work

While powerful, raw automated extraction still falls short of the near-perfect precision required for an encyclopedia, necessitating the human-in-the-loop component. Additionally, Wikipedia editors often create attribute names on the fly, leading to messy schemata that require constant cleaning. Future directions include expanding multi-lingual alignment of information and using the resulting "semantified Wikipedia" to power advanced question-answering systems.

Summary updated

BibTeX
@inproceedings{weld2008intelligence,
  author = {Daniel S. Weld and Fei Wu and Eytan Adar and Saleema Amershi and James Fogarty and Raphael Hoffmann and Kayur Patel and Michael Skinner},
  title = {{Intelligence in Wikipedia}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2008},
}
Figure