Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

Dan Su, Kezhi Kong, Ying Lin, Joseph Jennings, Brandon Norick, Markus Kliegl, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro

TL;DR

NVIDIA researchers have released Nemotron-CC, a massive 6.3-trillion-token dataset derived from Common Crawl that challenges the current trend of aggressive data filtering. By combining classifier ensembling, synthetic rephrasing, and smarter heuristic management, they created a dataset with 4x more unique tokens than competitors like DCLM, enabling an 8B model to significantly outperform Llama 3.1 8B on key benchmarks like MMLU.

What Problem Does This Paper Address?

As Large Language Models (LLMs) scale toward training on 15 trillion tokens or more (e.g., Llama 3.1), they are hitting a "data wall." Recent high-quality datasets like FineWeb-Edu and DCLM use aggressive model-based filtering to extract only the "best" web pages, often discarding 90% of the raw data.

While this creates high-quality subsets for short training runs, it leaves very few unique tokens for massive, long-horizon training. If a dataset only has 1 trillion unique tokens, a 15-trillion-token training run must repeat that data 15 times, leading to diminishing returns and inferior model performance. The authors aim to bridge this gap: how do we maintain high quality while preserving the massive quantity of unique data needed for the next generation of LLMs?

Methods

The authors move away from static, "blunt-force" filtering toward a more nuanced, learned pipeline:

  1. Optimized Extraction: They switched from the popular Trafilatura extractor to Justext, which yielded 28.6% more high-quality tokens from the same raw HTML.
  2. Ensemble Quality Labeling: Instead of relying on a single classifier (which might have a narrow "preference"), they ensembled three different models: a FineWeb-Edu-style classifier (educational value), a DCLM-style classifier (informativeness), and their own Nemotron-based models. This significantly increased the "recall" of high-quality documents.
  3. Selective Filtering: They discovered that traditional heuristic filters (like removing pages with too many stop words) often kill high-quality data. They disabled these filters for high-quality splits, applying them only to low-quality data.
  4. Synthetic Data Generation (SDG):
    • For Low-Quality Data: They used a medium-sized model to rephrase noisy web text into a clean, Wikipedia-like style.
  5. For High-Quality Data: They generated diverse Question-Answer (QA) pairs, summaries, and knowledge lists to "expand" the information and provide fresh, unique tokens.
Table 3: Synthetic data token count statistics (billion).

Datasets

The primary contribution is Nemotron-CC, which consists of:

  • 4.4 Trillion globally deduplicated original tokens.
  • 1.9 Trillion synthetically derived tokens.
  • Total: 6.3 Trillion tokens.

For comparison, they also evaluated a 1.1T high-quality subset called Nemotron-CC-HQ.

Dataset sizes in trillions of tokens. "Unique" shows the estimated number of tokens after global fuzzy deduplication of the real tokens.

Key Insights and Ideas

  • Quantity is a Quality of its own: Aggressive filtering is a "short-term gain, long-term pain" strategy. For massive training horizons, the "repetition penalty" of small, high-quality datasets outweighs the benefits of their purity.
  • The Power of the Ensemble: Different classifiers have different "blind spots." By ensembling, the authors boosted the high-quality token yield from 9% to 25% of the total crawl without sacrificing downstream accuracy.
  • Refining vs. Removing: Instead of just deleting "bad" data, rephrasing it (SDG) allows the model to learn from the underlying information without being distracted by web noise, HTML artifacts, or poor grammar.
  • Heuristics are Blunt Tools: The finding that disabling traditional filters for high-quality data actually improves results suggests that the community may have been over-filtering for years.

Key Results

The results demonstrate that Nemotron-CC is superior for both short and long training runs:

  • Short Horizon (1T tokens): The high-quality subset (Nemotron-CC-HQ) achieved an MMLU of 59.0, a massive +5.6 point jump over DCLM (53.4).
  • Long Horizon (15T tokens): An 8B model trained using 7.2T tokens from Nemotron-CC reached an MMLU of 70.3, significantly outperforming Llama 3.1 8B (65.3) and showing better average performance across ten diverse tasks.
Figure 1: MMLU scores for 8B parameter models trained for 1T tokens. Compared to DCLM, our meth- ods enable us to either create a 4× larger dataset of similar quality or increase the MMLU using a high qual- ity subset of the tokens. Having a larger dataset, in the sense of unique real tokens, is crucial when training over long horizons such as 15T tokens.

Limitations and Future Work

  • Language Scope: The current study and dataset are limited to English. Adapting these learned pipelines to other languages is a key next step.
  • Factual Integrity: The authors acknowledge they did not explicitly verify the factual accuracy of the rephrased synthetic data, which carries a risk of introducing hallucinations.
  • Compute Constraints: Due to resource limits, they did not rephrase "medium-quality" data, which represents a large portion of the crawl and could potentially be improved further.
  • Decontamination: The dataset was not explicitly decontaminated against benchmark test sets, though the authors note that their main competitors (FineWeb, DCLM) and Llama 3.1 also face similar contamination issues.

Summary updated

Abstract

Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html
BibTeX
@inproceedings{su2025nemotron,
  author = {Dan Su and Kezhi Kong and Ying Lin and Joseph Jennings and Brandon Norick and Markus Kliegl and Mostofa Patwary and Mohammad Shoeybi and Bryan Catanzaro},
  title = {{Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2025},
}
Figure