A Genre-based Clustering Approach to Content Extraction

Suhit Gupta, Hila Becker, Gail E. Kaiser, Salvatore J. StolfoColumbia 2005 · DOI

TL;DR

This paper introduces a genre-based clustering approach to automatically tailor content extraction settings for webpages. By clustering websites into genres using their text and search engine snippets, the system can dynamically apply optimized filter settings, significantly improving content extraction for applications like mobile browsing and accessibility tools.

What Problem Does This Paper Address?

The web is rife with "clutter" – elements like menus, ads, and animations – that obscure the actual content of a webpage. This makes it difficult for users to access information, especially on devices with limited screen real estate (like PDAs or cell phones) or for visually impaired users relying on screen readers. While content extraction tools exist, they often rely on generic settings that don't perform optimally across diverse website types. The challenge is to automatically adapt these extraction settings to the specific "genre" or type of website being viewed, without requiring manual configuration for every new site.

Methods

The core of the approach is a two-stage process:

  1. Offline Genre Clustering:
    • Data Augmentation: For each website in a large corpus, the system collects its own textual content and "snippets" (short descriptions) obtained by querying popular search engines with the website's domain name. This snippet data is crucial for increasing the frequency of descriptive function words.
  2. Feature Representation: A "Word Key" vector is created, representing the frequency of significant words across all documents. Each website's content is then mapped to this Word Key, creating a frequency graph.
  3. Clustering: Manhattan distance is used to measure the similarity between these frequency graphs. Hierarchical clustering is then applied to group similar websites into genres. This method doesn't require pre-defining the number of clusters.
  4. Online Content Extraction:
    • Classification: When a new, unclassified webpage is encountered, its Word Key graph is generated. The system then efficiently (in linear time) finds the closest pre-classified genre cluster.
  5. Setting Application: The content extraction filter settings previously optimized for that genre are automatically applied to the webpage. If no sufficiently close cluster is found, default settings are used.

The system's framework, "Crunch," acts as a web proxy that parses HTML into a Document Object Model (DOM) tree, applies filters based on the genre-specific settings, and then serves the cleaned HTML to the user.

Datasets

The primary dataset used was a corpus of 171 unique websites. These were chosen to represent a variety of genres, including news (international and regional), shopping, astronomy, and technical blogs. The paper does not introduce a new benchmark dataset but rather uses this collection for evaluating their clustering and content extraction approach.

Key Insights and Ideas

  • Snippets are Gold: Leveraging search engine snippets significantly improves the accuracy of genre classification. These snippets provide crucial context and boost the frequency of words that effectively distinguish website types.
  • Genre Matters for Extraction: Website designers often employ similar page layouts and HTML structures within the same genre. This means that content extraction settings optimized for one news site, for example, will likely work well for other news sites.
  • Unsupervised Genre Discovery: The hierarchical clustering approach allows the system to discover genres organically, without needing to pre-specify the number or types of genres. This is vital for the vast and ever-changing landscape of the web.
  • Efficiency is Key: The offline clustering process is computationally intensive (O(n²)), but the online classification of new websites is very fast (linear time), making it practical for real-time web browsing.

Key Results

  • Improved Clustering Accuracy: Using snippets resulted in finding more clusters (14 vs. 5) and a higher percentage of accurately categorized sites (87.5% vs. 31%) compared to not using snippets.
  • Significant Content Extraction Improvements:
    • Speech Rendering: For visually impaired users, Crunch reduced reading time by 10-80% compared to raw webpages, with the greatest improvements on subsidiary article pages.
  • Constrained Devices: On PDAs, Crunch increased the amount of content displayed on the first screenful by an average of 215% (and up to 750% for news articles). Similar improvements were seen on cell phones.
  • Robustness: The approach showed resilience to changes in website structure over time, as snippets tend to remain more stable than full page content.

Limitations and Future Work

  • A Priori Settings Required: The system relies on having pre-configured "good settings" for at least some genres or nearby clusters. If no such settings exist, it falls back to default settings, which may not be optimal.
  • Unmatched Genres: The system can identify websites that don't fit into any existing cluster. While this is a strength, it means these sites will use default settings.
  • Future Work: The authors suggest exploring a broader range of website pre-classification to cover more genres and potentially developing mechanisms for deconflicting competing (different) settings if multiple genres apply. They also note that further research is needed in this area, particularly for websites that are already W3C-compliant.

Summary updated

Abstract

The content of a webpage is usually contained within a small body of text and images, or perhaps several articles on the same page; however, the content may be lost in the clutter (defined as cosmetic features such as animations, menus, sidebars, obtrusive banners). Automatic content extraction has many applications, including browsing on small cell phone and PDA screens, speech rendering for the visually impaired, and reducing noise for information retrieval systems. We have developed a framework, Crunch, which employs various heuristics for content extraction in the form of filters applied to the webpage's DOM tree; the filters aim to prune or transform the clutter, leaving only the content. Crunch allows users to tune what we call 'settings', consisting of thresholds for applying a particular filter and/or for toggling a filter on/off, because the HTML components that characterize clutter can vary significantly from website to website. However, we have found that the same settings tend to work well across different websites of the same genre, e.g., news or shopping, since the designers often employ similar page layouts. In particular, Crunch could obtain the settings for a previously unknown website by automatically classifying it as sufficiently similar to a cluster of known websites with previously adjusted settings. We present our approach to clustering a large corpus of websites into genres, using their pre-extraction textual material augmented by the snippets generated by searching for the website's domain name in web search engines. Including these snippets increases the frequency of function words needed for clustering. We use existing Manhattan distance measure and hierarchical clustering techniques, with some modifications, to pre-classify the corpus into genres offline. Our method does not require prior knowledge of the set of genres that websites fit into, but to be useful a priori settings must be available for some member of each cluster or a nearby cluster (otherwise defaults are used). Crunch classifies newly encountered websites online in linear-time, and then applies the corresponding filter settings, with no noticeable delay added by our content-extracting web proxy.
BibTeX
@inproceedings{gupta2005genre,
  author = {Suhit Gupta and Hila Becker and Gail E. Kaiser and Salvatore J. Stolfo},
  title = {{A Genre-based Clustering Approach to Content Extraction}},
  booktitle = {Columbia},
  year = {2005},
  doi = {10.7916/d87p99m9},
}
Figure