Target-Aware Language Modeling via Granular Data Sampling

Ernie Chang, Pin-Jie Lin, Yang Li, Changsheng Zhao, Daeil Kim, Rastislav Rabatin, Zechun Liu, Yangyang Shi, Vikas Chandra

Abstract

Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows selecting large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance while preserving its effectiveness on other tasks. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with ~1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.

Introduction

Modern Large Language Models (LLMs) are typically trained on trillions of tokens sourced from across the web. While this broad exposure enables general-purpose capabilities, it is computationally expensive and may not yield optimal performance for specific, niche domains. The central challenge lies in data selection: identifying a small, representative "coreset" that enhances performance on target tasks without sacrificing general knowledge. Previous selection methods often relied on simple word-based features, which can be too coarse to capture task-specific nuances or overly biased, leading to a loss in general-purpose utility.

Methodology: Multi-granular Tokenization and Sampling

The proposed approach introduces a new paradigm for importance sampling based on multi-granular tokenization. This method strikes a balance between fine-grained subwords and coarse-grained phrases to better represent document content.

Overview of the multi-granular tokenization and data sampling pipeline.

The pipeline consists of four primary stages:

  1. Tokenizer Adaptation: The standard Llama-3 tokenizer is merged with tokens learned specifically from target task data. This creates a vocabulary containing subwords, words, and multi-word phrases.
  2. Feature Extraction: Documents are converted into fixed-size feature vectors (Doc2Vec) using hashed N-grams. The multi-granular nature of the tokens makes these vectors more robust against information loss and hash collisions.
  3. Importance Weighting: The significance of each document in a large raw pool (e.g., RefinedWeb) is calculated relative to the target task distribution. Documents that align closely with the target receive higher weights.
  4. Sampling: The top $k$ documents are selected based on these weights to form the final training coreset.

A key finding was the "magic mix" of token types: an optimal distribution of approximately 60% subwords, 30% words, and 10% multi-word tokens. This ratio prevents the model from becoming overly specialized while retaining the necessary detail for specific tasks.

Experimental Results

The effectiveness of the multi-granular sampling approach was evaluated on eight common-sense reasoning and NLP benchmarks, including ARC, BoolQ, and HellaSwag, using model sizes ranging from 125M to 1.5B parameters.

Efficiency and Scaling

One of the most significant results is that models trained on just 1% of the original data performed on par with those trained on the full RefinedWeb dataset. Compared to random sampling, the multi-granular approach yielded a 6.94% average improvement across benchmarks.

Accuracy comparison across different model sizes (125M to 1.5B) for multi-granular, n-gram, and random sampling.

As shown in the scaling analysis, the performance gains are consistent as model size increases. The multi-granular method consistently outperforms both random sampling and standard n-gram-based selection.

Task Robustness

Unlike traditional importance sampling, which can cause performance degradation on non-target tasks, the multi-granular approach maintains general knowledge. This is attributed to the inclusion of coarser-grained tokens (phrases) that help preserve broader linguistic context. The researchers observed a strong correlation (Pearson $r = 0.82$) between the reduction in KL divergence to the target distribution and the resulting downstream accuracy.

Conclusion and Future Work

This work demonstrates that intelligent data selection via multi-granular tokenization can drastically reduce the data requirements for pretraining effective language models. By using only 1% of the data, models can achieve performance comparable to full-scale training.

Future research will explore whether these findings scale to massive models (70B+ parameters) and investigate the computational overhead of featurizing billions of documents. Additionally, the authors aim to integrate explicit bias mitigation strategies into the sampling process to ensure the safety and fairness of the resulting models.

Summary updated

Abstract

Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows selecting large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance *while preserving its effectiveness on other tasks*. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with ~1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.
BibTeX
@inproceedings{chang2024target,
  author = {Ernie Chang and Pin-Jie Lin and Yang Li and Changsheng Zhao and Daeil Kim and Rastislav Rabatin and Zechun Liu and Yangyang Shi and Vikas Chandra},
  title = {{Target-Aware Language Modeling via Granular Data Sampling}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=TAQWem7tTK},
}
Figure