TL;DR
This paper introduces LASER, a novel, efficient, and comprehensive pipeline for selecting high-quality instruction-tuning data for Large Language Models (LLMs). By intelligently scoring data for difficulty and quality, and employing stratified sampling with clustering for diversity, LASER significantly improves LLM performance while reducing the need for massive datasets.
What Problem Does This Paper Address?
Large Language Models (LLMs) are powerful tools, but their ability to perform a wide range of tasks is heavily influenced by the data used for instruction tuning (IT). While it's known that IT datasets can be downsampled without significant performance loss, existing data selection methods often struggle with:
- High computational costs: Many methods rely on large, closed-weight LLM judges, making them expensive.
- Narrow domain focus: Approaches are often limited to specific task types or domains, hindering their generalizability.
- Incomplete criteria: Most methods address only one or two of the key IT data properties: difficulty, quality, and diversity.
- Brittleness: They can fail to outperform simple heuristics like random sampling or selecting the longest responses in real-world scenarios.
This paper aims to address these limitations by proposing a robust, complete, and efficient data selection pipeline that works across diverse tasks and models.
Methods
LASER employs a three-stage pipeline:
- Instruction Classification: A lightweight classifier is trained to categorize instructions into seven distinct types (Math, Coding, Generation, Reasoning, Brainstorming, Factual QA, Extraction). This allows for task-specific scoring.
- Scoring:
- Difficulty Scoring: A general-purpose difficulty scorer is trained using a heterogeneous pool of 18 LLMs. It predicts how likely an average model is to incorrectly solve a data point, independent of its category. This is achieved by analyzing deviations in model performance across datasets.
- Quality Scoring: Dedicated quality scorers are designed for each instruction category. For example, a process reward model is used for Math, and a code-quality scorer for Coding. These scorers assess the usefulness and accuracy of responses.
- Combined Scoring: Difficulty ($f$) and quality ($q$) scores are combined into an overall preference score ($p = f \times q$).
- Clustering + Ranking: To ensure diversity and avoid redundancy, LASER uses a stratified sampling approach. It first determines a fixed quota for each category and then clusters data points within each category using k-means. Finally, it selects the highest-scoring sample from each cluster, with a mechanism to discard clusters with very low scores.
Datasets
The paper uses an aggregation of several datasets for training and evaluation, including:
- Source Datasets (for training D): HuggingFaceH4/ifeval-like-data, vicgalle/alpaca-gpt4, nvidia/OpenMathInstruct-2, ai2-adapt-dev/flan_v2_converted, openbmb/UltraInteract_sft (Coding), WizardLMTeam/WizardLM_evol_instruct_V2_196K, theblackcat102/sharegpt-english, microsoft/orca-agentinstruct-1M-v1 (random subset).
- Evaluation Benchmarks: A comprehensive suite including GSM8k, HumanEval, MBPP, AlpacaEval, IFeval, ARC-C, BBH, GPQA, Hellaswag, MuSR, Winogrande, AGIEval, MMLU, TruthfulQA, OpenBookQA.
The paper also introduces a skewed distribution of datasets ($D_{skewed}$) to simulate homogeneous data and uses $D_{strong}$ (Tülu v3) and $D_{weak}$ (aggregation of less effective datasets) to test robustness.
Key Insights and Ideas
- Holistic Approach is Key: The paper strongly emphasizes that combining difficulty, quality, and diversity is crucial for effective data selection. Relying on any single criterion alone yields suboptimal results.
- Task-Specific Quality Matters: Generic quality scorers are insufficient. Tailoring quality assessment to specific task types (e.g., correctness for Math/Coding, constraint adherence for Generation) leads to better outcomes.
- Difficulty as a Measure of "Learning Signal": The difficulty scorer is designed to predict how likely a model is to fail an instruction. This is a more informative signal for learning than simply identifying easy or hard tasks.
- Stratified Sampling + Clustering for Diversity: Simply selecting high-scoring samples can lead to redundancy. LASER's approach of stratifying by category and then clustering within categories effectively balances diversity and quality.
- Efficiency Gains: Despite its multi-faceted approach, LASER is computationally efficient, especially compared to methods that scale poorly with dataset size.
Key Results
- Performance Gains: LASER consistently outperforms baselines (random sampling, longest response, DEITA, CAR) across various benchmarks and model sizes, achieving significant score gains over base models. For instance, with 100k samples, LASER surpasses full-data fine-tuning on some benchmarks.
- Robustness: LASER demonstrates strong performance across different model families (Mistral, Llama, Falcon, Qwen, SmolLM) and sizes, as well as across diverse source dataset distributions ($D_{strong}$, $D_{weak}$, $D_{skewed}$).
- Bias Mitigation: In skewed data distributions, LASER effectively reduces category bias, whereas other diversity-aware methods like DEITA can exacerbate it.
- Efficiency: LASER's runtime remains stable even with increasing sample sizes, making it more practical than methods like CAR which struggle with large datasets.
Limitations and Future Work
- Difficulty Scorer Reliability: The authors acknowledge potential unreliability in LLM-based evaluations, including prompt formatting issues and LLM biases. While they mitigate this, exact difficulty scores might not perfectly reflect a model's true capacity.
- Quality Scorer Expansion: Dedicated quality scorers for Reasoning, Factual QA, and Extraction were not developed. The authors suggest that Process Reward Models (PRMs) could be adapted for these tasks.
- Weak Source Data Performance: For very weak source data, LASER shows modest gains, suggesting a potential ceiling effect. In such cases, iterative data refinement guided by reliable scorers might be more beneficial.
- IRT-based Difficulty: While IRT-based difficulty targets showed higher validity with external criteria (human difficulty scores), they did not improve data selection performance. This is noted as an exciting direction for future research.
- Modular Design: The modular nature of LASER allows for future improvements to individual scorers.
Summary updated