Introduction
The performance of modern Large Language Models (LLMs) is increasingly tied to the quality of their pretraining data rather than architectural innovations alone. However, web-scale corpora, often comprising trillions of tokens, are notoriously noisy and unstructured. While high-quality data is desirable, manual annotation is unfeasible at scale, and using state-of-the-art models like GPT-4 for massive data engineering tasks is prohibitively expensive. This paper introduces DecorateLM, a framework that distills data engineering expertise into a compact 1.2-billion parameter model to efficiently "decorate" raw corpora with quality metadata.
Methodology: The Three Pillars of Decoration
DecorateLM transforms raw text into a high-quality "Decorated Corpus" through a three-phase strategy:
- Rating: The model scores text across eight dimensions, such as educational value, reasoning level, and scarcity, allowing for nuanced data selection.
- Tagging: A three-level hierarchical system (e.g., Natural Sciences → Physics → Modern Physics) categorizes text to ensure domain diversity and targeted upsampling.
- Editing: Noisy web content is rephrased into a formalized, "Wikipedia-style" format, significantly reducing perplexity and improving learnability.
Efficiency through Distillation
The core innovation lies in using a Small Language Model (SLM) as a specialized data engineer. The authors used GPT-4 to annotate a small seed dataset of 30,000 samples. A 1.2B parameter MiniCPM model was then trained on this data to mimic GPT-4’s assessments. Surprisingly, this distilled SLM often provides more continuous and consistent quality ratings than its teacher, as it specializes in the specific task of data evaluation rather than general-purpose generation.
Experimental Results
The framework was applied to 100 billion tokens from major open-source datasets (Dolma, C4, The Pile, CC-CN, and Baidu Wiki). From this, a 45-billion-token "Decorated Corpus" was sampled. Training a 1.2B parameter model on this refined subset yielded significant gains:
- Mathematical Reasoning: Scores on the GSM8K benchmark rose from 38.9 to 45.0.
- Domain Coverage: Hierarchical tagging enabled the upsampling of rare fields, leading to a 7.5-point increase on a specialized Domain Coverage (DC) benchmark.
- Dataset Insights: The analysis revealed that English datasets like Dolma and The Pile generally exhibit higher quality and diversity than large-scale Chinese web crawls.
Conclusion and Limitations
DecorateLM demonstrates that small, specialized models can effectively scale high-quality data engineering. While the framework significantly boosts model performance, it may inherit biases from the teacher model (GPT-4) and currently focuses primarily on English and Chinese. Future work will explore scaling these gains to larger models and expanding the language and tagging granularity.
Summary updated