Abstract
The World Wide Web contains an immense volume of structured data embedded within HTML <table> tags. However, a significant portion of this data remains inaccessible to automated systems because tables are frequently used for visual layout rather than data storage. This paper presents a framework for automatically identifying "genuine" data tables and determining their internal structure. By employing string similarity metrics—such as Levenshtein distance and n-grams—and machine learning classifiers, we demonstrate a robust pipeline for transforming neglected web content into structured Linked Open Data.
Introduction
While search engines excel at indexing unstructured text and multimedia, the structured data locked inside billions of HTML tables often remains "dark." It is estimated that the web contains over 600 billion facts within these tables. To unlock this potential for the Semantic Web, machines must be able to distinguish between layout-only tables (e.g., navigation bars) and genuine data tables, and subsequently interpret their orientation and schema.
Methodology
Our approach integrates a multi-step workflow into the Information Workbench platform, focusing on two primary challenges: genuineness detection and orientation classification.
String Metrics and Heuristics
We utilize three primary string similarity metrics to compare cell contents: Levenshtein distance (edit distance), Jaro-Winkler (prefix matching), and n-grams (substring matching). Based on these metrics, we developed heuristics to calculate horizontal and vertical similarity traces. High vertical similarity typically indicates that a column represents a specific attribute (e.g., a list of dates), suggesting a vertical table orientation.
Machine Learning Classification
We evaluated four standard machine learning algorithms—Naive Bayes, Decision Trees, k-nearest neighbors (kNN), and Support Vector Machines (SVM)—to process these similarity traces. A key optimization, referred to as the "modified" approach in our evaluations, involved "equalizing" numeric values. By treating different numbers as the same symbol, the algorithms can more easily recognize numeric columns regardless of specific values.
Evaluation and Results
The system was trained and tested on a dataset of 400 tables extracted from the 2008 WebTables corpus.
Orientation Detection
The determination of table orientation (horizontal vs. vertical) was highly successful. Most classifiers, particularly Decision Trees and SVMs, achieved near 100% accuracy when using Levenshtein distance as the underlying metric, especially when employing the numeric equalization modification.
Genuineness Filtering
Distinguishing "real" data from layout tables proved more challenging. The kNN algorithm, when combined with n-grams similarity, yielded the best results, achieving an F-measure of approximately 0.78-0.79. As shown in the results, the modified approach consistently outperformed the unmodified baseline across most classifiers.
Overall, the integrated system achieves a total performance of approximately 77%, successfully extracting knowledge from three out of every four arbitrary tables encountered.
Conclusion and Future Work
This research provides a scalable method for surfacing structured data from the "neglected" corners of the web. While orientation detection is nearly solved, future work will focus on improving genuineness classification through more complex structural parameters and fully automating ontology mapping to link extracted data to knowledge bases like DBpedia and Wikidata.
Summary updated