TL;DR
NVIDIA researchers have released Nemotron-CC, a massive 6.3-trillion-token dataset derived from Common Crawl that challenges the current trend of aggressive data filtering. By combining classifier ensembling, synthetic rephrasing, and smarter heuristic management, they created a dataset with 4x more unique tokens than competitors like DCLM, enabling an 8B model to significantly outperform Llama 3.1 8B on key benchmarks like MMLU.
What Problem Does This Paper Address?
As Large Language Models (LLMs) scale toward training on 15 trillion tokens or more (e.g., Llama 3.1), they are hitting a "data wall." Recent high-quality datasets like FineWeb-Edu and DCLM use aggressive model-based filtering to extract only the "best" web pages, often discarding 90% of the raw data.
While this creates high-quality subsets for short training runs, it leaves very few unique tokens for massive, long-horizon training. If a dataset only has 1 trillion unique tokens, a 15-trillion-token training run must repeat that data 15 times, leading to diminishing returns and inferior model performance. The authors aim to bridge this gap: how do we maintain high quality while preserving the massive quantity of unique data needed for the next generation of LLMs?
Methods
The authors move away from static, "blunt-force" filtering toward a more nuanced, learned pipeline:
- Optimized Extraction: They switched from the popular
Trafilaturaextractor toJustext, which yielded 28.6% more high-quality tokens from the same raw HTML. - Ensemble Quality Labeling: Instead of relying on a single classifier (which might have a narrow "preference"), they ensembled three different models: a FineWeb-Edu-style classifier (educational value), a DCLM-style classifier (informativeness), and their own Nemotron-based models. This significantly increased the "recall" of high-quality documents.
- Selective Filtering: They discovered that traditional heuristic filters (like removing pages with too many stop words) often kill high-quality data. They disabled these filters for high-quality splits, applying them only to low-quality data.
- Synthetic Data Generation (SDG):
- For Low-Quality Data: They used a medium-sized model to rephrase noisy web text into a clean, Wikipedia-like style.
- For High-Quality Data: They generated diverse Question-Answer (QA) pairs, summaries, and knowledge lists to "expand" the information and provide fresh, unique tokens.
Datasets
The primary contribution is Nemotron-CC, which consists of:
- 4.4 Trillion globally deduplicated original tokens.
- 1.9 Trillion synthetically derived tokens.
- Total: 6.3 Trillion tokens.
For comparison, they also evaluated a 1.1T high-quality subset called Nemotron-CC-HQ.
Key Insights and Ideas
- Quantity is a Quality of its own: Aggressive filtering is a "short-term gain, long-term pain" strategy. For massive training horizons, the "repetition penalty" of small, high-quality datasets outweighs the benefits of their purity.
- The Power of the Ensemble: Different classifiers have different "blind spots." By ensembling, the authors boosted the high-quality token yield from 9% to 25% of the total crawl without sacrificing downstream accuracy.
- Refining vs. Removing: Instead of just deleting "bad" data, rephrasing it (SDG) allows the model to learn from the underlying information without being distracted by web noise, HTML artifacts, or poor grammar.
- Heuristics are Blunt Tools: The finding that disabling traditional filters for high-quality data actually improves results suggests that the community may have been over-filtering for years.
Key Results
The results demonstrate that Nemotron-CC is superior for both short and long training runs:
- Short Horizon (1T tokens): The high-quality subset (Nemotron-CC-HQ) achieved an MMLU of 59.0, a massive +5.6 point jump over DCLM (53.4).
- Long Horizon (15T tokens): An 8B model trained using 7.2T tokens from Nemotron-CC reached an MMLU of 70.3, significantly outperforming Llama 3.1 8B (65.3) and showing better average performance across ten diverse tasks.
Limitations and Future Work
- Language Scope: The current study and dataset are limited to English. Adapting these learned pipelines to other languages is a key next step.
- Factual Integrity: The authors acknowledge they did not explicitly verify the factual accuracy of the rephrased synthetic data, which carries a risk of introducing hallucinations.
- Compute Constraints: Due to resource limits, they did not rephrase "medium-quality" data, which represents a large portion of the crawl and could potentially be improved further.
- Decontamination: The dataset was not explicitly decontaminated against benchmark test sets, though the authors note that their main competitors (FineWeb, DCLM) and Llama 3.1 also face similar contamination issues.
Summary updated