FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering

Erik Henriksson, Otto Tarkka, Filip GinterNodalida/baltic-hlt 2025

TL;DR

This paper introduces FinerWeb-10BT, a novel LLM-based line-level filtering method to improve the quality of web data used for training large language models (LLMs). By using GPT-40 mini to label individual lines and then training a DeBERTa-v3 classifier, they demonstrate that this approach significantly enhances model performance and training efficiency, even with reduced dataset sizes.

What Problem Does This Paper Address?

Training large language models (LLMs) requires massive datasets, and while the size of these datasets has exploded, so has the focus on data quality. Traditional methods for cleaning web data often rely on simple heuristics that can either miss low-quality content or, worse, accidentally remove valuable text. This leads to suboptimal LLM performance and increased computational costs. Furthermore, the environmental impact of training these models is a growing concern, making efficient data cleaning crucial for both better and greener AI. The paper aims to address the limitations of existing heuristic filters by proposing a more precise, LLM-driven approach to identify and remove low-quality text at a granular, line-by-line level.

Methods

The core of this work is an LLM-driven data filtering pipeline.

  1. LLM-based Line-Level Labeling: The authors first used GPT-40 mini to label a sample of 20,000 documents from the FineWeb dataset at the line level. Instead of predefined categories, GPT-40 mini generated descriptive labels for low-quality lines, allowing for a data-driven taxonomy. These initial labels were then refined and grouped into nine broader categories.
  2. Classifier Training: A DeBERTa-v3 classifier was trained on this LLM-annotated data. This classifier is designed to scale the line-level quality assessment to a much larger dataset (10B tokens of FineWeb).
  3. Platt Scaling: To address potential overconfidence in the classifier, Platt scaling was applied to adjust predicted probabilities, ensuring a more accurate reflection of true quality.
  4. Evaluation: The effectiveness of the filtering was evaluated by training GPT-2 models on both the original and filtered datasets and comparing their performance on the HellaSwag benchmark.

Datasets

  • FineWeb: The primary dataset used is FineWeb, a large collection of English text. The authors specifically worked with a 10B-token subset called FineWeb-10BT.
  • FinerWeb-10BT: This is the quality-annotated dataset introduced by the authors, containing line-level quality labels generated by their LLM-based pipeline.
  • HellaSwag: This benchmark was used to evaluate the performance of LLMs trained on the cleaned and original datasets.

Key Insights and Ideas

  • LLMs as Granular Quality Assessors: The paper demonstrates that LLMs can effectively identify subtle low-quality text at the line level, going beyond the capabilities of traditional heuristic filters. This allows for a more precise cleaning process.
  • Data-Driven Labeling: Allowing the LLM to generate its own descriptive labels for low-quality content, rather than relying on pre-defined categories, leads to a more flexible and comprehensive understanding of data quality issues.
  • Efficiency Gains: Cleaner data not only leads to better model performance but also significantly reduces training time and, by extension, energy consumption. This has important implications for both cost-effectiveness and environmental sustainability in AI development.
  • Bias Towards Keeping Data: The authors intentionally biased their classifier to err on the side of labeling potentially low-quality lines as "Clean" rather than misclassifying high-quality lines. This minimizes the risk of discarding valuable training data.

Key Results

  • Improved Model Performance: GPT-2 models trained on the FinerWeb-10BT dataset (even with up to 25% data reduction) achieved higher accuracy on the HellaSwag benchmark compared to models trained on the original FineWeb data.
  • Faster Training: Models trained on the cleaned data reached target accuracy significantly faster. For instance, the 25% reduced dataset showed a 32% reduction in training time (approximately 1 hour and 45 minutes saved per training round).
  • High Agreement with Human Annotators: The LLM-generated labels showed moderate to substantial agreement with human annotators, as measured by Cohen's Kappa scores (average of 0.70 for the full label set).
  • Effective Classifier: The DeBERTa-v3 classifier achieved strong performance in distinguishing between high- and low-quality lines, with F1 scores for the "Clean" class reaching up to 0.90.

Limitations and Future Work

  • English-Centric: The current study focuses solely on English documents. The authors acknowledge that LLM performance can be worse in low-resource languages, and care must be taken before generalizing these results to multilingual datasets.
  • LLM Bias: The LLM's inherent biases (e.g., over-sensitivity to certain language) can influence labeling. Manual adjustments were made, but further work is needed to mitigate these biases.
  • Vagueness of "Quality": The definition of "high-quality" data is inherently subjective and can be vague, leading to some noise in the data.
  • Future Work: The authors plan to:
  • Broaden the labeling scheme for a more comprehensive description of document content.
  • Experiment with more nuanced filtering approaches beyond simple score-based thresholds.
  • Compare their method against baselines like random data reduction.
  • Test their approach with different LLM architectures (e.g., Llama-style models).
  • Extend the method to other datasets and languages.

Summary updated

BibTeX
@inproceedings{henriksson2025finerweb,
  author = {Erik Henriksson and Otto Tarkka and Filip Ginter},
  title = {{FinerWeb-10BT: Refining Web Data with LLM-Based Line-Level Filtering}},
  booktitle = {Nodalida/baltic-hlt},
  year = {2025},
}
Figure