The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only

Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, Julien Launay

TL;DR

The Falcon LLM team challenges the conventional wisdom that large language models require labor-intensive, curated datasets to achieve high performance. By introducing a rigorous processing pipeline called Macrodata Refinement (MDR), they demonstrate that properly filtered and aggressively deduplicated web data alone can train models that match or exceed the performance of those trained on curated corpora. To prove it, they release RefinedWeb, a 5-trillion-token web-only dataset, along with a public 600-billion-token extract and competitive 1.3B/7.5B parameter models.

What Problem Does This Paper Address?

As large language models (LLMs) scale up, they require trillions of tokens for optimal pretraining. Traditionally, AI researchers have relied on a mix of raw web scrapes and curated "high-quality" datasets (like books, scientific papers, and conversational forums) to ensure broad zero-shot generalization.

However, human curation does not scale. Curating data requires specialized pipelines for every source, presents legal and licensing hurdles, and risks running into a "data wall" where unique, high-quality human text is exhausted. If web data is inherently inferior—as common consensus has held—scaling LLMs further becomes a massive bottleneck. This paper investigates whether web data can be salvaged through superior processing alone, eliminating the need for manual curation.

Methods

The authors developed the Macrodata Refinement (MDR) pipeline to transform raw CommonCrawl data into a pristine, pretraining-ready corpus. The pipeline operates in three main phases:

  1. Document Preparation: Instead of using pre-extracted text files (WET) from CommonCrawl, which often contain messy navigation menus and ads, the authors pull raw HTML (WARC) and use the trafilatura library for high-quality text extraction. They apply strict URL filtering using a blocklist of 4.6 million domains alongside a subword-based scoring system to filter out adult and fraudulent content.
  2. Filtering: They implement document-wise heuristics (checking symbol-to-word ratios, line lengths, and repetitions) to eliminate machine-generated spam. Crucially, they introduce line-wise corrections to scrub remaining boilerplate like "social media counters" or "navigation buttons" without discarding the entire document.
  3. Aggressive Deduplication: This is the secret sauce. The authors combine fuzzy deduplication (using MinHash LSH with highly aggressive settings to catch templated pages) and exact substring deduplication (using suffix arrays to remove duplicate sequences longer than 50 tokens across different documents).
Figure 2. Subsequent stages of Macrodata Refinement remove nearly 90% of the documents originally in CommonCrawl. Notably, filtering and deduplication each result in a halving of the data available: around 50% of documents are discarded for not being English, 24% of remaining for being of insufficient quality, and 12% for being duplicates. We report removal rate (grey) with respect to each previous stage, and kept rate (shade) overall. Rates measured in % of documents in the document preparation phase, then in tokens.

Datasets

  • CommonCrawl: The raw starting material spanning dumps from 2008 to 2023.
  • RefinedWeb: The resulting dataset containing 5 trillion tokens (~10 billion documents). The authors publicly release a 600-billion-token random extract.
  • Baselines for Comparison: The authors benchmark against prominent massive web datasets (C4, OSCAR) and curated datasets (The Pile, GPT-3's private training mix, and PaLM's dataset).
Comparison of RefinedWeb against existing massive web and curated datasets.

Key Insights and Ideas

  • Curation is Not a Silver Bullet: Meticulously filtered web data can actually outperform curated datasets like The Pile. Web data is not inherently lower quality; it is simply noisier by default.
  • Deduplication Matters More Than Filtering: While document filtering heuristics show variable success across different datasets, stringent deduplication consistently boosts zero-shot downstream performance across the board.
  • Avoid ML Filters for Quality: Unlike other pipelines that use lightweight classifiers trained on "gold standard" data (like Wikipedia) to score web pages, the authors stick to simple, neutral heuristics. This avoids introducing algorithmic biases that might over-filter text from minority groups or niche topics.

Key Results

To evaluate the data quality, the authors trained 1B and 3B parameter models to optimality, as well as full-scale 1B and 7B parameter models on 350 billion tokens.

  • Outperforming Curated Corpora: Models trained exclusively on RefinedWeb significantly outperformed publicly available models trained on The Pile (such as Pythia and GPT-NeoX).
  • Matching Private SOTA: At equivalent compute budgets, the RefinedWeb-trained models matched the zero-shot performance of the original GPT-3 models.
Figure 1. Models trained on REFINEDWEB alone outperform models trained on curated corpora. Zero-shot performance on our main-agg task aggregate (see Section 4.1 for details). At equivalent compute budgets, our models significantly outperform publicly available models trained on ▼The Pile, and match the performance of the ■GPT-3 models when tested within our evaluation setup.

Limitations and Future Work

  • Toxicity and Bias: Using Perspective API, the authors found RefinedWeb's toxicity distribution to be similar to The Pile. However, simple keyword and URL filtering do not fully address deeper social biases, which require further quantitative study.
  • Multi-Epoch Tradeoffs: In data-constrained scenarios, researchers often repeat datasets over multiple epochs. The authors note that while deduplication reduces the total token count, it appears to mitigate the performance degradation typically seen when training over multiple epochs. Further research is needed to explore this trade-off.
  • Multilingual Scaling: The current public release focuses strictly on English. While the MDR pipeline is largely language-agnostic, specific filters (like line-wise corrections) require careful per-language tuning to avoid over-filtering.

Summary updated

Abstract

Large language models are commonly trained on a mixture of filtered web data and curated high-quality corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it.
BibTeX
@inproceedings{penedo2023refinedweb,
  author = {Guilherme Penedo and Quentin Malartic and Daniel Hesslow and Ruxandra Cojocaru and Alessandro Cappelli and Hamza Alobeidli and Baptiste Pannier and Ebtesam Almazrouei and Julien Launay},
  title = {{The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=kM5eGcdCzq},
}
Figure