IBEX: Harvesting Entities from the Web Using Unique Identifiers

Aliaksandr Talaika, Joanna Biega, Antoine Amarilli, Fabian M. Suchanek

Abstract

IBEX (id-based entity extraction) introduces a novel approach to large-scale web data harvesting by leveraging unique identifiers—such as ISBNs, GTINs, and DOIs—as anchors for extraction. By analyzing how names cluster around these IDs across billions of web pages, the system automatically filters out noise and builds a massive, high-accuracy database of millions of unique entities. This method bypasses the limitations of traditional wrapper induction and named entity recognition, proving highly effective at capturing the "long tail" of web data.

What Problem Does This Paper Address?

The Web is a massive but messy repository of information. Traditional Named Entity Recognition (NER) can identify that a string like "Samsung Galaxy S4" is a product, but it struggles with two major hurdles:

  1. Disambiguation: If a page lists a product name, a manufacturer, and a "See Also" link, which one actually belongs to the product ID listed?
  2. Structural Decay: Many web pages use broken HTML or non-semantic layouts. Standard "wrapper induction" methods—which rely on consistent DOM structures—fail when applied to the diverse, "wild" web.
Figure 2: A typical web product listing where multiple candidate names (Manufacturer, Model, Brand) surround a unique identifier, complicating extraction.

The authors tackle the challenge of harvesting entities at "Web scale" (hundreds of millions of pages) without needing pre-defined templates or perfectly formatted data.

Methods

The IBEX pipeline operates in three distinct phases, moving from local extraction to global statistical cleaning:

  • Phase 1: Frame Tree Extraction. Instead of relying on the often-broken standard DOM tree, the authors developed Frame Trees. This method segments a page based on visual and logical separators (like <h1>, <p>, or sequences of <br>). It identifies "records"—regions of a page containing exactly one unique ID—and collects all nearby candidate names.
  • Phase 2: Global Outlier Detection. This is the core innovation. The system looks at the distribution of names across the entire web corpus. If a word like "Price" appears associated with thousands of different IDs, it is flagged as noise. However, if a specific name consistently appears with one specific ID and rarely elsewhere, it is identified as the true name of the entity.
  • Phase 3: Final Selection. For each unique ID, the system aggregates all surviving candidate names from across the web and selects the most frequent and highest-scoring one.
Figure 3: The variety of unique identifier types and the corresponding entities targeted by the IBEX system.

Datasets and Evaluation

The researchers utilized the ClueWeb09 and ClueWeb12 corpora, representing a massive snapshot of the web (~1.2 billion pages, 35 TB of data). To verify quality, they created a "gold standard" by manually annotating 200 random samples for each entity type (GTIN, CAS, DOI, and Email) and compared results against the YAGO knowledge base to estimate coverage.

Key Results and Insights

The system successfully extracted a massive database of unique entities with impressive precision:

  • Quantity: 13.6 million email addresses, 2.5 million commercial products (GTINs), 1 million documents (DOIs), and 235,000 chemical substances (CAS).
  • Accuracy: Final precision reached 82% for products, 73% for documents, and 96% for chemical substances.
  • Knowledge Discovery: IBEX found 1.4 million books that were completely absent from the YAGO knowledge base, demonstrating its ability to capture the "long tail" of web data.
  • IDs as Ground Truth: While names are ambiguous, unique identifiers are designed to be globally unique. By anchoring the search to the ID first, the problem of entity resolution is largely solved from the start.

Limitations and Future Work

The authors acknowledge that email addresses are "pseudo-IDs" because one person can have multiple emails, meaning the system cannot guarantee a 1:1 mapping for people as it does for products. Future work may extend the system to extract other attributes, such as prices for products or molecular formulas for chemicals, and expand the crawl scope to include major retailers like Amazon to further increase product coverage.

Summary updated

BibTeX
@inproceedings{talaika2015ibex,
  author = {Aliaksandr Talaika and Joanna Biega and Antoine Amarilli and Fabian M. Suchanek},
  title = {{IBEX: Harvesting Entities from the Web Using Unique Identifiers}},
  booktitle = {Webdb},
  year = {2015},
  doi = {10.1145/2767109.2767116},
}
Figure