TL;DR
This paper introduces a genre-based clustering approach to automatically tailor content extraction settings for webpages. By clustering websites into genres using their text and search engine snippets, the system can dynamically apply optimized filter settings, significantly improving content extraction for applications like mobile browsing and accessibility tools.
What Problem Does This Paper Address?
The web is rife with "clutter" – elements like menus, ads, and animations – that obscure the actual content of a webpage. This makes it difficult for users to access information, especially on devices with limited screen real estate (like PDAs or cell phones) or for visually impaired users relying on screen readers. While content extraction tools exist, they often rely on generic settings that don't perform optimally across diverse website types. The challenge is to automatically adapt these extraction settings to the specific "genre" or type of website being viewed, without requiring manual configuration for every new site.
Methods
The core of the approach is a two-stage process:
- Offline Genre Clustering:
- Data Augmentation: For each website in a large corpus, the system collects its own textual content and "snippets" (short descriptions) obtained by querying popular search engines with the website's domain name. This snippet data is crucial for increasing the frequency of descriptive function words.
- Feature Representation: A "Word Key" vector is created, representing the frequency of significant words across all documents. Each website's content is then mapped to this Word Key, creating a frequency graph.
- Clustering: Manhattan distance is used to measure the similarity between these frequency graphs. Hierarchical clustering is then applied to group similar websites into genres. This method doesn't require pre-defining the number of clusters.
- Online Content Extraction:
- Classification: When a new, unclassified webpage is encountered, its Word Key graph is generated. The system then efficiently (in linear time) finds the closest pre-classified genre cluster.
- Setting Application: The content extraction filter settings previously optimized for that genre are automatically applied to the webpage. If no sufficiently close cluster is found, default settings are used.
The system's framework, "Crunch," acts as a web proxy that parses HTML into a Document Object Model (DOM) tree, applies filters based on the genre-specific settings, and then serves the cleaned HTML to the user.
Datasets
The primary dataset used was a corpus of 171 unique websites. These were chosen to represent a variety of genres, including news (international and regional), shopping, astronomy, and technical blogs. The paper does not introduce a new benchmark dataset but rather uses this collection for evaluating their clustering and content extraction approach.
Key Insights and Ideas
- Snippets are Gold: Leveraging search engine snippets significantly improves the accuracy of genre classification. These snippets provide crucial context and boost the frequency of words that effectively distinguish website types.
- Genre Matters for Extraction: Website designers often employ similar page layouts and HTML structures within the same genre. This means that content extraction settings optimized for one news site, for example, will likely work well for other news sites.
- Unsupervised Genre Discovery: The hierarchical clustering approach allows the system to discover genres organically, without needing to pre-specify the number or types of genres. This is vital for the vast and ever-changing landscape of the web.
- Efficiency is Key: The offline clustering process is computationally intensive (O(n²)), but the online classification of new websites is very fast (linear time), making it practical for real-time web browsing.
Key Results
- Improved Clustering Accuracy: Using snippets resulted in finding more clusters (14 vs. 5) and a higher percentage of accurately categorized sites (87.5% vs. 31%) compared to not using snippets.
- Significant Content Extraction Improvements:
- Speech Rendering: For visually impaired users, Crunch reduced reading time by 10-80% compared to raw webpages, with the greatest improvements on subsidiary article pages.
- Constrained Devices: On PDAs, Crunch increased the amount of content displayed on the first screenful by an average of 215% (and up to 750% for news articles). Similar improvements were seen on cell phones.
- Robustness: The approach showed resilience to changes in website structure over time, as snippets tend to remain more stable than full page content.
Limitations and Future Work
- A Priori Settings Required: The system relies on having pre-configured "good settings" for at least some genres or nearby clusters. If no such settings exist, it falls back to default settings, which may not be optimal.
- Unmatched Genres: The system can identify websites that don't fit into any existing cluster. While this is a strength, it means these sites will use default settings.
- Future Work: The authors suggest exploring a broader range of website pre-classification to cover more genres and potentially developing mechanisms for deconflicting competing (different) settings if multiple genres apply. They also note that further research is needed in this area, particularly for websites that are already W3C-compliant.
Summary updated