DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models

Ranchi Zhao, Zhen Leng Thai, Yifan Zhang, Shengding Hu, Jie Zhou, Yunqi Ba, Jie Cai, Zhiyuan Liu, Maosong Sun

Introduction

The performance of modern Large Language Models (LLMs) is increasingly tied to the quality of their pretraining data rather than architectural innovations alone. However, web-scale corpora, often comprising trillions of tokens, are notoriously noisy and unstructured. While high-quality data is desirable, manual annotation is unfeasible at scale, and using state-of-the-art models like GPT-4 for massive data engineering tasks is prohibitively expensive. This paper introduces DecorateLM, a framework that distills data engineering expertise into a compact 1.2-billion parameter model to efficiently "decorate" raw corpora with quality metadata.

Methodology: The Three Pillars of Decoration

DecorateLM transforms raw text into a high-quality "Decorated Corpus" through a three-phase strategy:

  1. Rating: The model scores text across eight dimensions, such as educational value, reasoning level, and scarcity, allowing for nuanced data selection.
  2. Tagging: A three-level hierarchical system (e.g., Natural Sciences → Physics → Modern Physics) categorizes text to ensure domain diversity and targeted upsampling.
  3. Editing: Noisy web content is rephrased into a formalized, "Wikipedia-style" format, significantly reducing perplexity and improving learnability.
The DecorateLM pipeline: Distilling GPT-4 expertise into a Small Language Model (SLM) to process and refine raw corpora.

Efficiency through Distillation

The core innovation lies in using a Small Language Model (SLM) as a specialized data engineer. The authors used GPT-4 to annotate a small seed dataset of 30,000 samples. A 1.2B parameter MiniCPM model was then trained on this data to mimic GPT-4’s assessments. Surprisingly, this distilled SLM often provides more continuous and consistent quality ratings than its teacher, as it specializes in the specific task of data evaluation rather than general-purpose generation.

Rating accuracy: The distilled DecorateLM model demonstrates higher Spearman correlation with ground truth ratings compared to GPT-4.

Experimental Results

The framework was applied to 100 billion tokens from major open-source datasets (Dolma, C4, The Pile, CC-CN, and Baidu Wiki). From this, a 45-billion-token "Decorated Corpus" was sampled. Training a 1.2B parameter model on this refined subset yielded significant gains:

  • Mathematical Reasoning: Scores on the GSM8K benchmark rose from 38.9 to 45.0.
  • Domain Coverage: Hierarchical tagging enabled the upsampling of rare fields, leading to a 7.5-point increase on a specialized Domain Coverage (DC) benchmark.
  • Dataset Insights: The analysis revealed that English datasets like Dolma and The Pile generally exhibit higher quality and diversity than large-scale Chinese web crawls.

Conclusion and Limitations

DecorateLM demonstrates that small, specialized models can effectively scale high-quality data engineering. While the framework significantly boosts model performance, it may inherit biases from the teacher model (GPT-4) and currently focuses primarily on English and Chinese. Future work will explore scaling these gains to larger models and expanding the language and tagging granularity.

Summary updated

Abstract

The performance of Large Language Models (LLMs) is substantially influenced by the pretraining corpus, which consists of vast quantities of unsupervised data processed by the models. Despite its critical role in model performance, ensuring the quality of this data is challenging due to its sheer volume and the absence of sample-level quality annotations and enhancements. In this paper, we introduce DecorateLM, a data engineering method designed to refine the pretraining corpus through data rating, tagging and editing. Specifically, DecorateLM rates texts against quality criteria, tags texts with hierarchical labels, and edits texts into a more formalized format. Due to the massive size of the pretraining corpus, adopting an LLM for decorating the entire corpus is less efficient. Therefore, to balance performance with efficiency, we curate a meticulously annotated training corpus for DecorateLM using a large language model and distill data engineering expertise into a compact 1.2 billion parameter small language model (SLM). We then apply DecorateLM to enhance 100 billion tokens of the training corpus, selecting 45 billion tokens that exemplify high quality and diversity for the further training of another 1.2 billion parameter LLM. Our results demonstrate that employing such high-quality data can significantly boost model performance, showcasing a powerful approach to enhance the quality of the pretraining corpus.
BibTeX
@inproceedings{zhao2024decoratelm,
  author = {Ranchi Zhao and Zhen Leng Thai and Yifan Zhang and Shengding Hu and Jie Zhou and Yunqi Ba and Jie Cai and Zhiyuan Liu and Maosong Sun},
  title = {{DecorateLM: Data Engineering through Corpus Rating, Tagging, and Editing with Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.83},
}
Figure