TL;DR
This paper introduces SampleMix, a novel sample-wise data mixing strategy for LLM pre-training that moves beyond traditional domain-wise approaches. By evaluating the quality and diversity of individual samples, SampleMix dynamically constructs a more globally diverse and higher-quality training dataset, leading to improved performance and significantly faster training.
What Problem Does This Paper Address?
Large language models (LLMs) are heavily influenced by the data they are trained on. Current methods for mixing data from different sources (domains) typically work top-down: they assign weights to entire domains and then sample uniformly from within those domains. This approach has two major drawbacks:
- Ignoring Inter-domain Overlaps: Datasets often share semantic similarities or topics across domains. Domain-wise methods fail to leverage these commonalities, potentially leading to a less diverse overall training set.
- Suboptimal Sample Distribution: Uniform sampling within a domain can lead to a suboptimal distribution, where high-quality, diverse samples might be under-represented, or low-quality samples might still be included unnecessarily.
These limitations can result in suboptimal LLM performance and inefficient training.
Methods
SampleMix proposes a bottom-up, sample-wise approach to data mixing. Instead of pre-defining domain proportions, it focuses on individual data points:
- Quality Evaluation: A sophisticated quality evaluator, trained using GPT-40 and an ordinal regression model, assesses each document across seven dimensions (e.g., clarity, coherence, accuracy, significance). This provides a fine-grained quality score for every sample.
- Diversity Evaluation: Data diversity is captured by first clustering document embeddings (using K-Means) and then evaluating each sample's contribution to cluster compactness and separation. This metric quantifies how unique a sample is within its cluster and how distinct that cluster is from others.
- Sampling Weight Calculation: The quality and diversity scores for each sample are combined using a weighting factor ($\alpha$) to create a unified sampling weight. This allows for a balance between prioritizing high-quality and diverse data.
- Dynamic Sampling: Based on these sample-specific weights and a target token budget, SampleMix dynamically determines the sampling frequency for each document. This ensures that the final training dataset is optimally constructed to meet the desired quality and diversity goals within the given budget.
Datasets
- Training & Evaluation: SlimPajama dataset (a cleaned and deduplicated version of RedPajama) is used for training and analysis.
- Additional Evaluation: Chinese Wanjuan dataset is used to test generalization.
- Baselines: The paper compares SampleMix against several existing data mixture methods, including Vanilla (natural distribution), DoReMi, CE, BiMIX-OPT, DOGE, and DML.
Key Insights and Ideas
- Inter-domain Overlap is Significant: The paper empirically demonstrates substantial overlap in clusters across different domains in the SlimPajama dataset, highlighting the inadequacy of purely domain-wise sampling.
- Sample-level Control is Crucial: Focusing on individual sample quality and diversity, rather than just domain-level statistics, allows for a more nuanced and effective control over the training data's composition.
- Quality and Diversity are Complementary: While both quality and diversity are important, the optimal balance (determined by the $\alpha$ parameter) can vary. The paper suggests that for datasets with already high inherent quality (like SlimPajama), a higher emphasis on diversity might be beneficial, whereas for lower-quality datasets, prioritizing quality is key.
- Adaptability to Token Budgets: SampleMix's dynamic sampling approach allows it to effectively adapt to different token budget constraints, a significant advantage over static domain-wise methods.
Key Results
- Superior Performance: SampleMix achieves the highest average accuracy across eight downstream tasks, outperforming all baseline methods.
- Training Efficiency: SampleMix requires 1.4x to 2.1x fewer training steps to reach baseline performance levels, achieving average baseline accuracy 1.9x faster.
- Generalization: SampleMix demonstrates robust generalization capabilities on larger models (8B parameters) and across different datasets (Chinese Wanjuan).
- Cost-Effectiveness: Despite its sophisticated evaluation, SampleMix's computational cost is significantly lower than training an 8B model, representing only 2.68% of the cost.
Limitations and Future Work
- Hyperparameter Tuning: While SampleMix is designed as a universal method, optimal hyperparameters (especially the $\alpha$ value) may vary across datasets. The paper provides a user recommendation for tuning.
- Future Work: The authors suggest incorporating model-perspective evaluation metrics and exploring code data mixing as future research directions.
Summary updated