TL;DR
BoilerNet is a neural sequence labeling model designed to strip "boilerplate" (ads, navigation, banners) from web pages to extract their primary content. By treating a web page as a sequence of text blocks and using a Bi-LSTM architecture, it eliminates the need for brittle, hand-crafted features and outperforms state-of-the-art models on modern web layouts.
What Problem Does This Paper Address?
Extracting the "meat" of a web page is a fundamental preprocessing step for everything from "reader views" in browsers to large-scale natural language processing. However, web pages are messy. The primary content is often buried under layers of CMS-generated boilerplate.
Existing solutions typically fall into two traps:
- Over-engineering: They rely on hundreds of hand-crafted features (e.g., text density, link ratios, DOM depth). These features are often tailored to specific time periods or site styles and fail to generalize as web design evolves.
- Data Scarcity: Labeled datasets for boilerplate removal are small and expensive to produce, making it hard to train robust, modern models.
The authors of this paper propose BoilerNet, a model that learns features automatically from raw HTML and text, leveraging the "locality of authoring"—the observation that relevant content is usually written in contiguous segments within the underlying code.
Methods
BoilerNet treats boilerplate removal as a sequence labeling task.
- Input Representation: The model divides a web page into a sequence of "text blocks" (leaf nodes in the DOM tree). Each block is represented by a sparse vector encoding two things:
- Structural Context: The path of HTML tags from the root to that specific leaf node (e.g.,
html > body > div > p). -
Textual Content: The actual words appearing in the block. To keep the model efficient, they only consider the $k$ most common tags and $l$ most common words.
-
Architecture: As shown in Figure 2, these sparse vectors are passed through an embedding layer to create dense representations. These are then fed into bidirectional Long Short-Term Memory (Bi-LSTM) layers. The Bi-LSTM is crucial because it allows the model to consider the context before and after a block to decide if it's content or noise. Finally, a fully connected layer with a sigmoid activation classifies each block as either content (+) or boilerplate (-).
Datasets
The authors evaluated BoilerNet on two benchmarks:
- CleanEval (2007): A classic but dated dataset of diverse websites.
- GoogleTrends-2017: A new dataset created by the authors to reflect modern web structures. They sampled 180 documents from the top results of trending Google queries in 2017, ensuring the benchmark includes current design practices and diverse genres.
Key Insights and Ideas
- Locality of Authoring vs. Rendering: While previous models tried to infer content based on how a page looks (visual rendering), BoilerNet focuses on how it is authored. Because developers tend to write article text in a single contiguous block of code, sequence models are naturally suited to find these boundaries.
- Featureless Learning: By using raw HTML tags and words, the model avoids the "feature fatigue" of previous state-of-the-art models like Web2Text, which used 128 hand-crafted features.
- Generalization with Minimal Data: The model performs remarkably well even when trained on very small subsets (e.g., 50-55 instances), suggesting that the structural patterns of web content are highly learnable.
Key Results
BoilerNet proved to be more consistent across different datasets than its competitors.
- CleanEval: BoilerNet performed on par with the state-of-the-art Web2Text, achieving an F1-score of 0.87 for the positive (content) class.
- GoogleTrends-2017: On this more modern dataset, BoilerNet significantly outperformed all baselines. It achieved a recall of 0.88 and an F1-score of 0.78 for content extraction, representing an 11% increase in recall over its closest competitors.
The authors also released a browser extension to demonstrate the model's real-world utility, allowing users to highlight primary content with a single click.
Limitations and Future Work
While BoilerNet reduces the need for manual features, it still faces the industry-wide challenge of labeled data scarcity. The authors noted that their initial attempts at "weak supervision" (using noisy, automatically generated labels) did not yield significant improvements. Future work could explore more sophisticated semi-supervised or self-supervised learning techniques to further reduce the reliance on human-annotated data. Additionally, while the model handles raw HTML, extremely dynamic, JavaScript-heavy pages that don't resolve into a standard DOM tree structure might still present challenges.
Summary updated