Introduction
The modern web is a vast repository of structured information—product listings, search results, and directories—often stored in databases but rendered via HTML templates. To leverage this data for applications like price comparison or meta-search engines, researchers use "wrappers" to extract information back into structured formats. Traditionally, this required manual labeling (Wrapper Induction) or brittle string-matching techniques. This paper introduces DEPTA (Data Extraction based on Partial Tree Alignment), an automated system that "reverse engineers" web pages to extract structured data without manual intervention.
The DEPTA Methodology
DEPTA moves beyond raw HTML strings, treating web pages as visual structures. It operates in three distinct stages:
- Data Record Identification: By utilizing browser rendering information, DEPTA builds a robust Document Object Model (DOM) tree. It identifies "data regions"—areas containing similar sub-trees—using an Enhanced Simple Tree Matching algorithm.
- Partial Tree Alignment: This is the core innovation. Unlike traditional global alignment, which fails when fields are missing (e.g., a product without a discount), partial tree alignment uses a "seed" tree (the most complete record) and merges others into it. This allows the system to handle optional fields and complex nesting.
- Pattern Generation: The aligned records form a "consensus" tree, which is used to generate tree-based regular expressions. These patterns serve as the final wrapper for scraping similar pages.
Experimental Evaluation
The researchers evaluated DEPTA using a dataset of 200 web pages from 142 different websites, primarily focusing on "list pages" containing multiple data objects.
Key Results
DEPTA demonstrated high accuracy, significantly outperforming previous string-based methods. The system was evaluated on two levels: finding the records themselves and identifying the individual data fields within those records.
As shown in the table above, the system achieved over 96% recall for record extraction and 98% for item extraction. This indicates that once a record is located, the partial tree alignment is exceptionally effective at parsing its constituent parts (e.g., price, name, image).
Discussion and Limitations
While DEPTA is highly effective for list-based pages, it faces challenges with extremely deep or irregular nesting. Furthermore, while it automates wrapper generation, the problem of "wrapper repair"—automatically updating a scraper when a site's layout changes significantly—remains an area for future research. By prioritizing visual rendering over messy HTML syntax, DEPTA provides a more resilient foundation for automated web data mining.
Summary updated