Boilerplate Removal using a Neural Sequence Labeling Model

Jurek Leonhardt, Avishek Anand, Megha KhoslaWWW 2020 · DOI · OpenReview

TL;DR

BoilerNet is a neural sequence labeling model designed to strip "boilerplate" (ads, navigation, banners) from web pages to extract their primary content. By treating a web page as a sequence of text blocks and using a Bi-LSTM architecture, it eliminates the need for brittle, hand-crafted features and outperforms state-of-the-art models on modern web layouts.

What Problem Does This Paper Address?

Extracting the "meat" of a web page is a fundamental preprocessing step for everything from "reader views" in browsers to large-scale natural language processing. However, web pages are messy. The primary content is often buried under layers of CMS-generated boilerplate.

Existing solutions typically fall into two traps:

  1. Over-engineering: They rely on hundreds of hand-crafted features (e.g., text density, link ratios, DOM depth). These features are often tailored to specific time periods or site styles and fail to generalize as web design evolves.
  2. Data Scarcity: Labeled datasets for boilerplate removal are small and expensive to produce, making it hard to train robust, modern models.

The authors of this paper propose BoilerNet, a model that learns features automatically from raw HTML and text, leveraging the "locality of authoring"—the observation that relevant content is usually written in contiguous segments within the underlying code.

Methods

BoilerNet treats boilerplate removal as a sequence labeling task.

  1. Input Representation: The model divides a web page into a sequence of "text blocks" (leaf nodes in the DOM tree). Each block is represented by a sparse vector encoding two things:
  2. Structural Context: The path of HTML tags from the root to that specific leaf node (e.g., html > body > div > p).
  3. Textual Content: The actual words appearing in the block. To keep the model efficient, they only consider the $k$ most common tags and $l$ most common words.

  4. Architecture: As shown in Figure 2, these sparse vectors are passed through an embedding layer to create dense representations. These are then fed into bidirectional Long Short-Term Memory (Bi-LSTM) layers. The Bi-LSTM is crucial because it allows the model to consider the context before and after a block to decide if it's content or noise. Finally, a fully connected layer with a sigmoid activation classifies each block as either content (+) or boilerplate (-).

The BoilerNet architecture featuring Bi-LSTM layers for sequence labeling.

Datasets

The authors evaluated BoilerNet on two benchmarks:

  • CleanEval (2007): A classic but dated dataset of diverse websites.
  • GoogleTrends-2017: A new dataset created by the authors to reflect modern web structures. They sampled 180 documents from the top results of trending Google queries in 2017, ensuring the benchmark includes current design practices and diverse genres.

Key Insights and Ideas

  • Locality of Authoring vs. Rendering: While previous models tried to infer content based on how a page looks (visual rendering), BoilerNet focuses on how it is authored. Because developers tend to write article text in a single contiguous block of code, sequence models are naturally suited to find these boundaries.
  • Featureless Learning: By using raw HTML tags and words, the model avoids the "feature fatigue" of previous state-of-the-art models like Web2Text, which used 128 hand-crafted features.
  • Generalization with Minimal Data: The model performs remarkably well even when trained on very small subsets (e.g., 50-55 instances), suggesting that the structural patterns of web content are highly learnable.

Key Results

BoilerNet proved to be more consistent across different datasets than its competitors.

  • CleanEval: BoilerNet performed on par with the state-of-the-art Web2Text, achieving an F1-score of 0.87 for the positive (content) class.
  • GoogleTrends-2017: On this more modern dataset, BoilerNet significantly outperformed all baselines. It achieved a recall of 0.88 and an F1-score of 0.78 for content extraction, representing an 11% increase in recall over its closest competitors.
Quantitative results on the GoogleTrends-2017 dataset, showing BoilerNet's superior performance in the positive class.

The authors also released a browser extension to demonstrate the model's real-world utility, allowing users to highlight primary content with a single click.

An example of the BoilerNet browser extension highlighting the main content of a Wikipedia page.

Limitations and Future Work

While BoilerNet reduces the need for manual features, it still faces the industry-wide challenge of labeled data scarcity. The authors noted that their initial attempts at "weak supervision" (using noisy, automatically generated labels) did not yield significant improvements. Future work could explore more sophisticated semi-supervised or self-supervised learning techniques to further reduce the reliance on human-annotated data. Additionally, while the model handles raw HTML, extremely dynamic, JavaScript-heavy pages that don't resolve into a standard DOM tree structure might still present challenges.

Summary updated

Abstract

The extraction of main content from web pages is an important task for numerous applications, ranging from usability aspects, like reader views for news articles in web browsers, to information retrieval or natural language processing. Existing approaches are lacking as they rely on large amounts of hand-crafted features for classification. This results in models that are tailored to a specific distribution of web pages, e.g. from a certain time frame, but lack in generalization power. We propose a neural sequence labeling model that does not rely on any hand-crafted features but takes only the HTML tags and words that appear in a web page as input. This allows us to present a browser extension which highlights the content of arbitrary web pages directly within the browser using our model. In addition, we create a new, more current dataset to show that our model is able to adapt to changes in the structure of web pages and outperform the state-of-the-art model.
BibTeX
@inproceedings{leonhardt2020boilerplate,
  author = {Jurek Leonhardt and Avishek Anand and Megha Khosla},
  title = {{Boilerplate Removal using a Neural Sequence Labeling Model}},
  booktitle = {The Web Conference},
  year = {2020},
  doi = {10.1145/3366424.3383547},
}
Figure