TL;DR
The Falcon LLM team challenges the conventional wisdom that large language models require labor-intensive, curated datasets to achieve high performance. By introducing a rigorous processing pipeline called Macrodata Refinement (MDR), they demonstrate that properly filtered and aggressively deduplicated web data alone can train models that match or exceed the performance of those trained on curated corpora. To prove it, they release RefinedWeb, a 5-trillion-token web-only dataset, along with a public 600-billion-token extract and competitive 1.3B/7.5B parameter models.
What Problem Does This Paper Address?
As large language models (LLMs) scale up, they require trillions of tokens for optimal pretraining. Traditionally, AI researchers have relied on a mix of raw web scrapes and curated "high-quality" datasets (like books, scientific papers, and conversational forums) to ensure broad zero-shot generalization.
However, human curation does not scale. Curating data requires specialized pipelines for every source, presents legal and licensing hurdles, and risks running into a "data wall" where unique, high-quality human text is exhausted. If web data is inherently inferior—as common consensus has held—scaling LLMs further becomes a massive bottleneck. This paper investigates whether web data can be salvaged through superior processing alone, eliminating the need for manual curation.
Methods
The authors developed the Macrodata Refinement (MDR) pipeline to transform raw CommonCrawl data into a pristine, pretraining-ready corpus. The pipeline operates in three main phases:
- Document Preparation: Instead of using pre-extracted text files (WET) from CommonCrawl, which often contain messy navigation menus and ads, the authors pull raw HTML (WARC) and use the
trafilaturalibrary for high-quality text extraction. They apply strict URL filtering using a blocklist of 4.6 million domains alongside a subword-based scoring system to filter out adult and fraudulent content. - Filtering: They implement document-wise heuristics (checking symbol-to-word ratios, line lengths, and repetitions) to eliminate machine-generated spam. Crucially, they introduce line-wise corrections to scrub remaining boilerplate like "social media counters" or "navigation buttons" without discarding the entire document.
- Aggressive Deduplication: This is the secret sauce. The authors combine fuzzy deduplication (using MinHash LSH with highly aggressive settings to catch templated pages) and exact substring deduplication (using suffix arrays to remove duplicate sequences longer than 50 tokens across different documents).
Datasets
- CommonCrawl: The raw starting material spanning dumps from 2008 to 2023.
- RefinedWeb: The resulting dataset containing 5 trillion tokens (~10 billion documents). The authors publicly release a 600-billion-token random extract.
- Baselines for Comparison: The authors benchmark against prominent massive web datasets (C4, OSCAR) and curated datasets (The Pile, GPT-3's private training mix, and PaLM's dataset).
Key Insights and Ideas
- Curation is Not a Silver Bullet: Meticulously filtered web data can actually outperform curated datasets like The Pile. Web data is not inherently lower quality; it is simply noisier by default.
- Deduplication Matters More Than Filtering: While document filtering heuristics show variable success across different datasets, stringent deduplication consistently boosts zero-shot downstream performance across the board.
- Avoid ML Filters for Quality: Unlike other pipelines that use lightweight classifiers trained on "gold standard" data (like Wikipedia) to score web pages, the authors stick to simple, neutral heuristics. This avoids introducing algorithmic biases that might over-filter text from minority groups or niche topics.
Key Results
To evaluate the data quality, the authors trained 1B and 3B parameter models to optimality, as well as full-scale 1B and 7B parameter models on 350 billion tokens.
- Outperforming Curated Corpora: Models trained exclusively on RefinedWeb significantly outperformed publicly available models trained on The Pile (such as Pythia and GPT-NeoX).
- Matching Private SOTA: At equivalent compute budgets, the RefinedWeb-trained models matched the zero-shot performance of the original GPT-3 models.
Limitations and Future Work
- Toxicity and Bias: Using Perspective API, the authors found RefinedWeb's toxicity distribution to be similar to The Pile. However, simple keyword and URL filtering do not fully address deeper social biases, which require further quantitative study.
- Multi-Epoch Tradeoffs: In data-constrained scenarios, researchers often repeat datasets over multiple epochs. The authors note that while deduplication reduces the total token count, it appears to mitigate the performance degradation typically seen when training over multiple epochs. Further research is needed to explore this trade-off.
- Multilingual Scaling: The current public release focuses strictly on English. While the MDR pipeline is largely language-agnostic, specific filters (like line-wise corrections) require careful per-language tuning to avoid over-filtering.
Summary updated