SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity

Xiangyu Xi, Deyang Kong, Jian Yang, Jiawei Yang, Zhengyu Chen, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei YeEMNLP 2025 · OpenReview

TL;DR

This paper introduces SampleMix, a novel sample-wise data mixing strategy for LLM pre-training that moves beyond traditional domain-wise approaches. By evaluating the quality and diversity of individual samples, SampleMix dynamically constructs a more globally diverse and higher-quality training dataset, leading to improved performance and significantly faster training.

What Problem Does This Paper Address?

Large language models (LLMs) are heavily influenced by the data they are trained on. Current methods for mixing data from different sources (domains) typically work top-down: they assign weights to entire domains and then sample uniformly from within those domains. This approach has two major drawbacks:

  1. Ignoring Inter-domain Overlaps: Datasets often share semantic similarities or topics across domains. Domain-wise methods fail to leverage these commonalities, potentially leading to a less diverse overall training set.
  2. Suboptimal Sample Distribution: Uniform sampling within a domain can lead to a suboptimal distribution, where high-quality, diverse samples might be under-represented, or low-quality samples might still be included unnecessarily.

These limitations can result in suboptimal LLM performance and inefficient training.

Methods

SampleMix proposes a bottom-up, sample-wise approach to data mixing. Instead of pre-defining domain proportions, it focuses on individual data points:

  1. Quality Evaluation: A sophisticated quality evaluator, trained using GPT-40 and an ordinal regression model, assesses each document across seven dimensions (e.g., clarity, coherence, accuracy, significance). This provides a fine-grained quality score for every sample.
  2. Diversity Evaluation: Data diversity is captured by first clustering document embeddings (using K-Means) and then evaluating each sample's contribution to cluster compactness and separation. This metric quantifies how unique a sample is within its cluster and how distinct that cluster is from others.
  3. Sampling Weight Calculation: The quality and diversity scores for each sample are combined using a weighting factor ($\alpha$) to create a unified sampling weight. This allows for a balance between prioritizing high-quality and diverse data.
  4. Dynamic Sampling: Based on these sample-specific weights and a target token budget, SampleMix dynamically determines the sampling frequency for each document. This ensures that the final training dataset is optimally constructed to meet the desired quality and diversity goals within the given budget.

Datasets

  • Training & Evaluation: SlimPajama dataset (a cleaned and deduplicated version of RedPajama) is used for training and analysis.
  • Additional Evaluation: Chinese Wanjuan dataset is used to test generalization.
  • Baselines: The paper compares SampleMix against several existing data mixture methods, including Vanilla (natural distribution), DoReMi, CE, BiMIX-OPT, DOGE, and DML.

Key Insights and Ideas

  • Inter-domain Overlap is Significant: The paper empirically demonstrates substantial overlap in clusters across different domains in the SlimPajama dataset, highlighting the inadequacy of purely domain-wise sampling.
  • Sample-level Control is Crucial: Focusing on individual sample quality and diversity, rather than just domain-level statistics, allows for a more nuanced and effective control over the training data's composition.
  • Quality and Diversity are Complementary: While both quality and diversity are important, the optimal balance (determined by the $\alpha$ parameter) can vary. The paper suggests that for datasets with already high inherent quality (like SlimPajama), a higher emphasis on diversity might be beneficial, whereas for lower-quality datasets, prioritizing quality is key.
  • Adaptability to Token Budgets: SampleMix's dynamic sampling approach allows it to effectively adapt to different token budget constraints, a significant advantage over static domain-wise methods.

Key Results

  • Superior Performance: SampleMix achieves the highest average accuracy across eight downstream tasks, outperforming all baseline methods.
  • Training Efficiency: SampleMix requires 1.4x to 2.1x fewer training steps to reach baseline performance levels, achieving average baseline accuracy 1.9x faster.
  • Generalization: SampleMix demonstrates robust generalization capabilities on larger models (8B parameters) and across different datasets (Chinese Wanjuan).
  • Cost-Effectiveness: Despite its sophisticated evaluation, SampleMix's computational cost is significantly lower than training an 8B model, representing only 2.68% of the cost.

Limitations and Future Work

  • Hyperparameter Tuning: While SampleMix is designed as a universal method, optimal hyperparameters (especially the $\alpha$ value) may vary across datasets. The paper provides a user recommendation for tuning.
  • Future Work: The authors suggest incorporating model-perspective evaluation metrics and exploring code data mixing as future research directions.

Summary updated

Abstract

Existing pretraining data mixing methods for large language models (LLMs) typically follow a domain-wise methodology, a top-down process that first determines domain weights and then performs uniform data sampling across each domain. However, these approaches neglect significant inter-domain overlaps and commonalities, failing to control the global diversity of the constructed training dataset. Further, uniform sampling within domains ignores fine-grained sample-specific features, potentially leading to suboptimal data distribution. To address these shortcomings, we propose a novel sample-wise data mixture approach based on a bottom-up paradigm. This method performs global cross-domain sampling by systematically evaluating the quality and diversity of each sample, thereby dynamically determining the optimal domain distribution. Comprehensive experiments across multiple downstream tasks and perplexity assessments demonstrate that SampleMix surpasses existing domain-based methods. Meanwhile, SampleMix requires 1.4x to 2.1x fewer training steps to achieve the baselines’ performance, highlighting the substantial potential of SampleMix to optimize pre-training data.
BibTeX
@inproceedings{xi2025samplemix,
  author = {Xiangyu Xi and Deyang Kong and Jian Yang and Jiawei Yang and Zhengyu Chen and Wei Wang and Jingang Wang and Xunliang Cai and Shikun Zhang and Wei Ye},
  title = {{SampleMix: A Sample-wise Pre-training Data Mixing Strategy by Coordinating Data Quality and Diversity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=nF8Nmwupif},
}
Figure