Leveraging Web-Crawled Data for High-Quality Fine-Tuning

Jing Zhou, Chenglin Jiang, Wei Shen, Xiao Zhou, Xiaonan HeEMNLP 2024 · DOI

Abstract

Researchers from Baidu have developed a novel method to transform noisy, "low-quality" web-crawled mathematical data into high-quality training sets without relying on expensive human annotation or proprietary models like GPT-4. By training a "format converter" model on automatically aligned data pairs, they achieved a significant performance boost on Chinese math benchmarks, allowing a 7B parameter model to outperform much larger models and even GPT-3.5.

Introduction

To make Large Language Models (LLMs) excel at specific tasks like mathematical reasoning, high-quality Supervised Fine-Tuning (SFT) data is essential. Currently, researchers typically rely on expensive human labor or distillation from proprietary models like GPT-4. However, GPT-4 often struggles with specific domains and languages, such as Chinese mathematics.

While the web is a vast repository of math problems, this data is often "noisy." Optical Character Recognition (OCR) errors and formatting issues (e.g., $3^2$ becoming 32) create semantic inaccuracies that lead models to learn incorrect logic. This paper addresses the challenge of recycling this "trash" web data into "gold" training data through a model-based cleaning approach.

Methodology

The authors propose a four-step pipeline to create a "Model-based Data Cleaning" system, as illustrated in the figure below:

Overview of the model-based data cleaning pipeline, showing the transition from noisy web data to cleaned SFT data via a re-generation model.
  1. Fuzzy Matching: A small "seed" dataset of high-quality math problems is aligned with a massive, messy web-crawled dataset. By matching questions or answers after stripping non-essential characters, the system creates <low-quality, high-quality> pairs.
  2. Training a Format Converter: A base LLM (such as Qwen or ChatGLM) is fine-tuned on these pairs to learn how to "translate" garbled web text into clean, LaTeX-formatted math.
  3. Mass Conversion: The trained converter is applied to the broader web-crawled dataset, transforming hundreds of thousands of noisy examples into a standardized format.
  4. Final Fine-Tuning: A fresh LLM is trained on the combined seed data and the newly "cleaned" web data.

The core intuition is that rewriting data is easier for an LLM than solving a problem from scratch; by using the noisy answer as a hint, the model can reconstruct the correct logic.

Experimental Setup

  • Seed Dataset: 84,095 manually annotated instances.
  • Web-Crawled Dataset: 573,960 instances collected from the web, containing significant formatting noise.
  • Evaluation Benchmarks: Ape210K (Chinese elementary math) and CMATH (LLM-specific Chinese math benchmark).

Results and Analysis

The proposed method improved model performance by an average of 9.4% on Chinese math problems compared to using only high-quality seed data. Notably, the 7B parameter model (Ours-Qwen1.5) achieved an average score of 80.8, surpassing GPT-3.5-Turbo (73.6), Claude-2 (76.7), and DeepSeek-Chat 67B (78.5).

Accuracy comparison across different data volumes and cleaning methods for ChatGLM and Qwen, demonstrating the superiority of the model-based SFT approach.

Key Insights

  • Semantic Formatting: In mathematics, formatting is semantic. Model-based cleaning restores the lost dimensionality of formulas (e.g., superscripts) that rule-based scripts cannot infer.
  • Base Model Advantage: Stronger base models (like Qwen) benefit more from this re-generation strategy, showing higher gains over rule-based cleaning compared to weaker models.
  • Scaling Laws: Performance gains followed a log-linear relationship; doubling the amount of cleaned SFT data consistently improved scores.

Limitations and Future Work

While effective, the method currently requires a high-quality "seed" dataset for initial fuzzy matching. Future work may explore completely unsupervised cleaning or the addition of a mathematical verifier to ensure the logical correctness of the re-generated data. Additionally, the authors suggest expanding this "rewriting" paradigm to other complex domains like legal or medical records where web data is abundant but poorly structured.

Summary updated

BibTeX
@inproceedings{zhou2024leveraging,
  author = {Jing Zhou and Chenglin Jiang and Wei Shen and Xiao Zhou and Xiaonan He},
  title = {{Leveraging Web-Crawled Data for High-Quality Fine-Tuning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.660},
}
Figure