Abstract
Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows to select large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance while preserving its effectiveness on other tasks. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with ~1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.
Introduction
Standard language model (LM) pretraining is traditionally a "one-size-fits-all" endeavor, often requiring trillions of tokens to ensure broad performance across diverse tasks. However, in many real-world scenarios, the goal is to develop a specialist model—one that excels in specific domains like medical reasoning or legal analysis—without sacrificing general "common sense."
Researchers from Meta, Virginia Tech, and Iowa State have addressed this challenge by developing a "multi-granular" data sampling technique. This approach allows language models to achieve state-of-the-art performance using only ~1% of the original pretraining data by creating high-quality "coresets" that maintain general-purpose capabilities while boosting domain-specific expertise.
The Challenge of Data Selection
The primary hurdle in specialized pretraining is domain bias. Simply selecting documents that closely resemble a target task often narrows the model's focus too much, causing performance on unrelated tasks to plummet. Existing methods, such as importance sampling, frequently rely on simple word-based features. These are often either too coarse to capture linguistic nuances or too computationally expensive to scale to the massive datasets required for modern LMs.
Methodology: Multi-Granular Importance Sampling
The authors propose a novel approach to importance sampling using a Multi-granular Tokenizer. The pipeline consists of three main stages:
- Tokenizer Adaptation: Starting with a base tokenizer (such as Llama-3), the system adapts it to a target task. Rather than just adding new words, it uses an optimization process based on "vocabulary utility"—an entropy-based metric—to find the optimal mix of subwords, words, and multi-word phrases.
- Doc2Vec via Hashing: To keep representations efficient, each document in a massive raw dataset (like RefinedWeb) is converted into a fixed-size feature vector. A hashing trick is applied to the multi-granular n-grams to maintain a compact footprint.
- Granular Sampling: The system calculates "importance weights" by comparing the feature distribution of the raw data to the target task data. This allows for the selection of a tiny, high-quality subset—a coreset—to train the model.
The Granularity Trade-off
A key insight of this work is the balance between fine-grained and coarse-grained tokens. Subwords (fine-grained) are excellent at capturing task-specific knowledge and reducing domain bias, but they can slow down the sampling process. Conversely, multi-word phrases (coarse-grained) compress sequences and speed up sampling but can be too rigid. The researchers found that a mixture—roughly 60% subwords, 30% words, and 10% multi-words—provides the ideal balance of representation and efficiency.
Key Results and Insights
Efficiency and Performance
The multi-granular sampling method demonstrated remarkable efficiency. Models trained on just 1% of the data performed on par with models trained on the full RefinedWeb dataset. Compared to random sampling, the proposed method achieved a 6.94% average improvement across eight common-sense reasoning benchmarks, including ARC-Easy, ARC-Challenge, BoolQ, PIQA, SIQA, HellaSwag, OpenBookQA, and WinoGrande.
Scalability and Bias Mitigation
The benefits of this approach held true across various model sizes, from 125M parameters up to 1.5B parameters. Crucially, the multi-granular approach mitigated the performance drops on non-target tasks typically seen with standard n-gram sampling, proving that it selects data that is more "universally useful."
Predictive Power of KL Divergence
The study identified a strong correlation (Pearson r = 0.82) between the reduction of KL divergence (a measure of the difference between data distributions) and eventual downstream task performance. This suggests that researchers can predict how well a model will perform before the resource-intensive training process even begins.
Limitations and Future Directions
While the results are promising for models up to 1.5B parameters, it remains to be seen if these gains persist for "frontier" models exceeding 70B parameters. Additionally, while the method reduces domain bias, the authors note that social or algorithmic biases inherent in web-crawled data were not explicitly evaluated. Future work will focus on extending this approach to even larger datasets and exploring its interaction with other model-based data selection techniques.
Summary updated