Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models

Minghao Wu, Thuy-Trang Vu, Lizhen Qu, Reza HafEMNLP 2024 · DOI

Abstract

Fine-tuning Large Language Models (LLMs) often involves training on heterogeneous and imbalanced datasets. Traditional heuristic-based data balancing methods, such as temperature scaling, are static and fail to adapt to the model's evolving needs during training. This paper introduces Mixture-of-Skills (MOS), a bi-level optimization framework that employs reinforcement learning to dynamically adjust dataset sampling probabilities. By utilizing novel reward signals based on transferability and relative difficulty, MOS achieves superior performance across multiple benchmarks and architectures, while providing up to a 2.2x increase in convergence speed compared to standard baselines.

Introduction

The performance of fine-tuned LLMs is heavily dependent on the composition of the training data. In practice, researchers often mix diverse datasets—ranging from mathematical reasoning to creative writing—which are frequently imbalanced in size and nature. The standard industry practice relies on manual heuristics like proportional or temperature-based sampling to prevent larger datasets from dominating the training process. However, these methods are static and require extensive hyperparameter tuning. They ignore the dynamic nature of the learning process, where a model's data requirements may shift as it acquires different skills. Mixture-of-Skills (MOS) addresses this by allowing the model to autonomously determine the most beneficial data mixture at any given step.

Methodology

MOS is structured as a bi-level optimization problem involving two primary components: a student LLM ($\theta$) and a teacher Scorer Network ($\psi$). The Scorer, a lightweight 2-layer MLP, learns to output sampling probabilities for the available datasets.

The Mixture-of-Skills (MOS) framework illustrating the interaction between the Scorer, the LLM, and the reward feedback loop.

The Scorer is optimized using the REINFORCE algorithm, guided by three distinct reward signals:

  • Transferability: This signal measures how well a dataset's features generalize to the broader training pool, calculated via cosine similarity of embeddings.
  • Difficulty: Rather than using raw perplexity—which can be misleadingly low for structured data like math—MOS focuses on the relative decrease in perplexity. This identifies datasets where the model is making the most significant learning progress.
  • Learning Trajectory: An Exponential Moving Average (EMA) is applied to these rewards to stabilize the sampling distribution and prevent erratic shifts during training.

Additionally, the authors developed MOSPEC, a variant tailored for task-specific specialization, which prioritizes data that specifically enhances a target skill.

Experimental Setup

The framework was evaluated using a curated collection of approximately 103K examples spanning four domains: Mathematics (MathInstruct), Medicine (MedInstruct), General Chat (ShareGPT), and NLP Tasks (P3). Performance was measured on the MMLU and MT-bench benchmarks across three model architectures: Qwen1.5-0.5B, Gemma-2B, and Llama-3-8B.

Results and Analysis

MOS consistently outperformed traditional heuristic baselines. For Llama-3-8B, the best MOS configuration improved the aggregate $\mu_{BOTH}$ score by 2.45 points over the strongest heuristic baseline. A critical finding was the deceptive nature of raw perplexity; models often show lower perplexity on complex math than on natural language due to structural predictability. By rewarding the drop in perplexity, MOS effectively prioritizes areas of active learning.

Efficiency is a core advantage of the MOS framework. While the additional computation for the Scorer adds roughly 20% to the per-step training time, this is more than compensated for by the accelerated convergence.

Training loss curves comparing MOS against heuristic baselines, demonstrating a 2.2x faster convergence rate.

In specialization tasks, MOSPEC proved that a strategic mixture of auxiliary data can lead to better mastery of a specific domain (e.g., mathematics) than training on domain-specific data alone.

Conclusion and Future Work

Mixture-of-Skills represents a significant step toward automated, dynamic data curriculum learning for LLMs. It effectively handles data heterogeneity and imbalance while significantly reducing training time. Future research will explore scaling MOS to handle a larger number of datasets and reducing the computational overhead of the reward calculation process.

Summary updated

BibTeX
@inproceedings{wu2024mixture,
  author = {Minghao Wu and Thuy-Trang Vu and Lizhen Qu and Reza Haf},
  title = {{Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.787},
}
Figure