Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions

Max Dallabetta, Conrad Dobberstein, Adrian Breiding, Alan AkbikACL 2024

TL;DR

This paper introduces FUNDUS, a novel news scraping library that prioritizes extraction quality over quantity by using bespoke, manually crafted extractors for each supported newspaper. This approach significantly outperforms generic scrapers in producing complete, artifact-free news articles, making it ideal for applications where data quality is paramount.

What Problem Does This Paper Address?

Extracting high-quality news articles from the web is a crucial first step for many Natural Language Processing (NLP) applications, from social analysis to training language models. The challenge lies in two main areas: first, identifying and downloading relevant articles from a vast number of online sources, and second, accurately extracting the core article content from the surrounding HTML, which is often cluttered with advertisements, navigation elements, and other boilerplate. Existing libraries typically rely on generic heuristics or machine learning models that aim for broad applicability but often sacrifice accuracy, leading to incomplete or malformed article text. This paper addresses the need for a tool that can reliably deliver clean, complete news article content, even if it means supporting a more curated set of sources.

Methods

FUNDUS tackles the problem by adopting a fundamentally different approach to content extraction. Instead of using generic methods, it employs bespoke, manually crafted extractors (parsers) for each supported online newspaper. This allows developers to meticulously tailor the extraction logic to the specific formatting of each website, optimizing for accuracy and completeness. The library also integrates both crawling (finding and downloading article URLs) and content extraction into a single, unified pipeline. This means users don't need to juggle multiple tools. For backward crawling, FUNDUS supports the massive CC-NEWS dataset, enabling the creation of large corpora from historical web archives.

Datasets

  • Custom Evaluation Dataset: The authors created a new dataset consisting of 80 news articles (5 from each of the 16 English-language publishers supported by FUNDUS at the time of writing). These articles were manually annotated with gold standard content.
  • CC-NEWS Dataset: For evaluating data potential and backward crawling, FUNDUS utilizes the CC-NEWS dataset, a large archive of WARC-formatted data containing millions of news articles dating back to 2016.

Key Insights and Ideas

  • Quality over Quantity: The core idea is that for many NLP tasks, the quality of the extracted text is far more important than the sheer volume of articles. Bespoke extractors, while requiring more initial effort, unlock significantly higher extraction quality.
  • Newspaper-Specific Formatting: Online newspapers, despite their diversity, often exhibit consistent internal formatting guidelines. FUNDUS leverages this by creating specific parsers for each, rather than trying to find a one-size-fits-all solution.
  • Unified Pipeline: Combining crawling and extraction into a single library simplifies the workflow for users, especially those less technically inclined.
  • Community-Driven Expansion: The library is designed to be extensible, with an MIT license and clear documentation to encourage community contributions of new newspaper parsers.

Key Results

  • Highest Extraction Quality: FUNDUS achieved an average ROUGE-LSum F1-score of 97.69%, significantly outperforming other prominent scraping libraries (e.g., Trafilatura at 89.81%, news-please at 85.81%).
  • Consistent Performance: FUNDUS exhibited a much lower standard deviation in its F1-scores across different publishers compared to generic scrapers, indicating more reliable performance.
  • Data Potential: FUNDUS can be used to crawl the CC-NEWS dataset, yielding over 2 million articles with extracted text bodies from 2020-2024.
  • Efficiency: For crawling 10,000 articles from 39 publishers, FUNDUS took approximately 549 seconds. Processing 2023 data from CC-NEWS (approx. 8.2 TB) took 2.1 hours.

Limitations and Future Work

  • Scalability Limitation: The primary limitation is the inherent lack of scalability across a vast number of online newspapers due to the manual effort required to create bespoke extractors. However, the authors argue that for quality-critical applications, this trade-off is acceptable.
  • Maintenance Overhead: Online newspapers can change their formatting, requiring ongoing maintenance of extractors. FUNDUS has an automated check to flag such changes.
  • Future Work: The authors plan to investigate semi-automatic methods to suggest extraction rules, which could help alleviate the manual effort and improve the scalability of adding new newspaper support. They also invite the community to contribute parsers.

Summary updated

BibTeX
@inproceedings{dallabetta2024fundus,
  author = {Max Dallabetta and Conrad Dobberstein and Adrian Breiding and Alan Akbik},
  title = {{Fundus: A Simple-to-Use News Scraper Optimized for High Quality Extractions}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure