Introduction
The Web Data Commons (WDC) offers a massive knowledge graph of product data extracted from the web, providing a unique opportunity for researchers and engineers to build e-commerce analytics without proprietary crawlers. However, in its raw form, this 100GB+ dataset is riddled with noise, bias, and scale-related challenges. This study conducts an empirical analysis of the 2018 WDC Product dataset to distill five best practices for extracting high-quality, reliable information from web-scale structured data.
The Challenge of Web-Scale Structured Data
While Schema.org has become the standard for embedding structured data into HTML, the resulting crawl is often "broad and expansive" but semantically inconsistent. Using this data "as-is" presents several risks:
- Noise and Skew: Misspellings, inaccurate language tags, and domain dominance can bias downstream models.
- Semantic Validity: Fields like "product name" often contain URLs or null strings instead of valid text.
- Scale: The sheer volume of data (billions of quads) makes manual cleaning impossible.
Empirical Methodology
The researchers performed four broad studies on the November 2018 WDC Microdata dataset, which contains approximately 4.8 billion quads across 7.4 million URLs:
- Language Analysis: Comparing explicit language tags (e.g., "en") against automated identification using fastText.
- Domain Trustworthiness: Using Google PageRank to identify high-quality sources versus low-quality spam sites.
- Semantic Heuristics: Developing manual rules to verify if fields contain expected data types (e.g., numeric ratings).
- Localization Study: Calculating the minimum "window size" needed to capture all triples for a single product node during sequential processing.
Key Findings
1. Language Tags are Unreliable
One of the most significant findings is the "non-trivial share of disagreement" between explicit metadata tags and actual content. For many languages, the provided tag does not match the detected language of the text literals. Practitioners cannot trust metadata alone; they must use automated identifiers to ensure data quality.
2. Semantic Validity is Property-Dependent
Data quality varies wildly depending on the specific Schema.org property. While product names and URLs are generally reliable, other fields are much noisier. For example, the study found that SKUs (Stock Keeping Units) had an invalidity rate of 38%.
3. The "Magic Number" for Processing
To handle this data without massive distributed computing clusters, the authors investigated sequential processing. They discovered that a sliding window of 145 triples is sufficient to capture 99% of all properties for any given product node. This finding allows for memory-efficient processing on standard hardware.
Conclusion and Best Practices
Based on these results, engineers looking to utilize WDC data should follow these four best practices:
- Verify Languages: Use tools like fastText rather than relying on HTML lang tags.
- Filter by Trust: Use PageRank or similar metrics to prioritize data from established domains.
- Apply Heuristic Checks: Implement length and type checks (e.g., ensuring ratings are numeric) before ingestion.
- Optimize Windows: Use a sliding window of 145 triples for efficient, sequential data extraction.
Summary updated