TL;DR
This paper introduces CETD, a novel DOM-based method for extracting the main content from web pages by analyzing "text density." It outperforms existing methods by accurately identifying and preserving the original structure of the content, making it valuable for various web mining applications.
What Problem Does This Paper Address?
The internet is awash with information, but web pages are not just pure content. They're cluttered with "noise" like navigation menus, advertisements, and copyright notices. This noise, which can constitute 40-50% of a page's data, significantly hinders web data mining and information retrieval. Traditional methods often struggle with the dynamic nature of web pages and the increasing complexity introduced by CSS. Existing approaches might fail to adapt to structural changes or require extensive manual effort for each website. The challenge is to efficiently and accurately extract the core content while preserving its original structure, enabling better downstream processing.
Methods
The core of the proposed method, Content Extraction via Text Density (CETD), lies in analyzing the "text density" of nodes within a Document Object Model (DOM) tree.
- Text Density (TD): This metric quantifies the importance of a DOM node by calculating the ratio of characters to tags within its subtree. Nodes with high TD are likely to contain substantial, simply formatted text (content), while those with low TD are more likely to be noise.
- Composite Text Density (CTD): To further refine the importance measure, CTD incorporates information about hyperlinks. It considers the proportion of hyperlink characters and tags relative to the total characters and tags, aiming to better distinguish content from noise, especially in pages with many links.
- DensitySum: This novel technique addresses the issue of content nodes being missed due to low individual text density or noise nodes being incorrectly identified due to high density. It works by summing the text densities of a node's children. The algorithm then identifies the node with the maximum DensitySum under the
<body>tag as the primary content block. This approach is more robust than simple thresholding or data smoothing. - DOM Tree Preservation: A key advantage is that CETD operates on the DOM tree, allowing it to preserve the original structure of the extracted content, unlike methods that simply output raw text.
Datasets
The authors used two primary sources for their experiments:
- CleanEval: This is a shared task and competitive evaluation dataset for cleaning arbitrary web pages. It includes English and Chinese divisions, with hand-labeled data. The authors noted some errors in the gold standard and manually re-extracted content for their experiments.
- Custom Datasets: To evaluate on varied sources, they created their own datasets:
- Big 5: Consisting of pages from Ars Technica, BBC, Yahoo!, New York Times, and Wikipedia.
- Chaos: Randomly selected pages from Google News and popular blog platforms (WordPress, Blogger). For these datasets, they arbitrarily selected 100 pages from each "Big 5" site and 200 from "Chaos," with content manually extracted and saved as UTF-8 text files to serve as the gold standard.
Key Insights and Ideas
- Text Density as a Proxy for Content: The fundamental idea that content is generally longer, less formatted, and has fewer hyperlinks than noise is a powerful heuristic.
- Hyperlinks as a Key Differentiator: Recognizing that hyperlinks are a significant component of "noise" and incorporating them into the density calculation (Composite Text Density) significantly improves accuracy.
- DensitySum's Robustness: The DensitySum technique is crucial for handling cases where individual nodes might have misleading densities. By aggregating child densities, it can identify larger, more coherent content blocks, even if some internal nodes have lower individual scores.
- DOM Structure Preservation is Valuable: The ability to retain the original HTML structure of the content, rather than just outputting plain text, opens up possibilities for more sophisticated downstream applications (e.g., rendering on small screens, structured data extraction).
- Unsupervised Learning: The method is unsupervised, meaning it doesn't require manually labeled training data for each website, making it highly scalable and adaptable.
Key Results
The proposed CETD method, particularly the CECTD-DS variant (Composite Text Density with DensitySum), demonstrated strong performance across various datasets.
- Overall Improvement: The average F1-score for CECTD-DS was 8.79% higher than the best scores among several alternative methods.
- Comparison with Alternatives: CECTD-DS consistently outperformed other methods like CETD-DS, CECTD-S, BTE, DSC, FE, KFE, LQ, and CETR across most datasets.
- CleanEval Performance: The CECTD-DS achieved scores higher than the winner of the CleanEval competition (84.1% F1-score on English data).
- Trade-offs: The paper highlights a precision-recall trade-off that can be managed by adjusting a threshold coefficient (λ), allowing users to prioritize recall if needed.
Limitations and Future Work
- Limitations:
- The method struggles with highly visual sites like video and picture platforms (e.g., YouTube), where it primarily extracts comments.
- Portal homepages, with their dense navigation menus and short news descriptions (often hyperlinks), also pose a challenge for accurate content identification.
- The current implementation relies on HTML Tidy, which sometimes fails to parse pages correctly.
- Future Work:
- Incorporate the WebKit kernel for more robust page parsing.
- Investigate identifying and handling "hidden" non-visible elements (like disclaimers) that might skew density calculations.
- Explore incorporating spatial information (position, size) into the density measure to further enhance performance.
- Extend experiments to compare with visual information-based methods like VIPS.
Summary updated