Abstract
Fine-tuning Large Language Models (LLMs) often involves training on heterogeneous and imbalanced datasets. Traditional heuristic-based data balancing methods, such as temperature scaling, are static and fail to adapt to the model's evolving needs during training. This paper introduces Mixture-of-Skills (MOS), a bi-level optimization framework that employs reinforcement learning to dynamically adjust dataset sampling probabilities. By utilizing novel reward signals based on transferability and relative difficulty, MOS achieves superior performance across multiple benchmarks and architectures, while providing up to a 2.2x increase in convergence speed compared to standard baselines.
Introduction
The performance of fine-tuned LLMs is heavily dependent on the composition of the training data. In practice, researchers often mix diverse datasets—ranging from mathematical reasoning to creative writing—which are frequently imbalanced in size and nature. The standard industry practice relies on manual heuristics like proportional or temperature-based sampling to prevent larger datasets from dominating the training process. However, these methods are static and require extensive hyperparameter tuning. They ignore the dynamic nature of the learning process, where a model's data requirements may shift as it acquires different skills. Mixture-of-Skills (MOS) addresses this by allowing the model to autonomously determine the most beneficial data mixture at any given step.
Methodology
MOS is structured as a bi-level optimization problem involving two primary components: a student LLM ($\theta$) and a teacher Scorer Network ($\psi$). The Scorer, a lightweight 2-layer MLP, learns to output sampling probabilities for the available datasets.
The Scorer is optimized using the REINFORCE algorithm, guided by three distinct reward signals:
- Transferability: This signal measures how well a dataset's features generalize to the broader training pool, calculated via cosine similarity of embeddings.
- Difficulty: Rather than using raw perplexity—which can be misleadingly low for structured data like math—MOS focuses on the relative decrease in perplexity. This identifies datasets where the model is making the most significant learning progress.
- Learning Trajectory: An Exponential Moving Average (EMA) is applied to these rewards to stabilize the sampling distribution and prevent erratic shifts during training.
Additionally, the authors developed MOSPEC, a variant tailored for task-specific specialization, which prioritizes data that specifically enhances a target skill.
Experimental Setup
The framework was evaluated using a curated collection of approximately 103K examples spanning four domains: Mathematics (MathInstruct), Medicine (MedInstruct), General Chat (ShareGPT), and NLP Tasks (P3). Performance was measured on the MMLU and MT-bench benchmarks across three model architectures: Qwen1.5-0.5B, Gemma-2B, and Llama-3-8B.
Results and Analysis
MOS consistently outperformed traditional heuristic baselines. For Llama-3-8B, the best MOS configuration improved the aggregate $\mu_{BOTH}$ score by 2.45 points over the strongest heuristic baseline. A critical finding was the deceptive nature of raw perplexity; models often show lower perplexity on complex math than on natural language due to structural predictability. By rewarding the drop in perplexity, MOS effectively prioritizes areas of active learning.
Efficiency is a core advantage of the MOS framework. While the additional computation for the Scorer adds roughly 20% to the per-step training time, this is more than compensated for by the accelerated convergence.
In specialization tasks, MOSPEC proved that a strategic mixture of auxiliary data can lead to better mastery of a specific domain (e.g., mathematics) than training on domain-specific data alone.
Conclusion and Future Work
Mixture-of-Skills represents a significant step toward automated, dynamic data curriculum learning for LLMs. It effectively handles data heterogeneity and imbalance while significantly reducing training time. Future research will explore scaling MOS to handle a larger number of datasets and reducing the computational overhead of the reward calculation process.
Summary updated