Abstract
Finetuning large language models with a variety of instruction-response pairs has enhanced their capability to understand and follow instructions. Current instruction tuning primarily relies on teacher models or human intervention to generate and refine the instructions and responses for training, which are costly, non-sustainable, and may lack diversity. In this paper, we introduce Mosaic Instruction Tuning (Mosaic-IT), a human/model-free compositional data augmentation method that can efficiently create rich and diverse augmentations from existing instruction tuning data to enhance the LLMs. Mosaic-IT randomly concatenates multiple instruction data into one and trains the model to produce the corresponding responses with predefined higher-level meta-instructions to strengthen its multi-step instruction-following and format-following skills. Our extensive evaluations demonstrate a superior performance and training efficiency of Mosaic-IT, which achieves consistent performance improvements over various benchmarks and a 80% reduction in training costs compared with original instruction tuning.
1. Introduction
Instruction Tuning (IT) is the critical process that transforms a raw base model into a helpful assistant. However, high-quality IT datasets are notoriously difficult to obtain, typically requiring either expensive human labor or "distillation" from massive teacher models like GPT-4. Furthermore, most existing IT datasets consist of simple, one-to-one instruction-response pairs. This creates a gap: real-world users often provide complex, multi-part queries, but models trained on simple data struggle to manage the interference and dependencies of multiple instructions at once. Mosaic-IT aims to bridge this gap without the need for new data generation.
2. Mosaic Instruction Tuning Strategies
The core idea of Mosaic-IT is inspired by "mosaic" data augmentation in computer vision. It applies four rule-based strategies to existing text data:
- Primary Mosaic Strategy: Randomly samples $k$ instruction-response pairs and concatenates them, teaching the model to handle denser contexts.
- Format Strategy: Introduces meta-instructions forcing specific response formats (e.g., tags or brackets) to enhance controllability.
- Permute Strategy: Requires the model to respond to concatenated instructions in a specific order (e.g., reverse order), increasing task complexity.
- Maskout Strategy: Instructs the model to ignore specific instructions within the mosaic, testing selective attention.
Crucially, these strategies are purely rule-based. No teacher model is needed to rewrite the text; the complexity is synthesized by simply rearranging and tagging existing data.
3. Experimental Evaluation
Mosaic-IT was validated across a wide array of popular open-source datasets (Alpaca, Alpaca-GPT4, WizardLM-70k, Magpie, and Vicuna-1M) and benchmarks including the Huggingface Open LLM Leaderboard, Alpaca Eval 2, MT-Bench, and IF Eval. Models tested included Mistral-7B, Llama-2 (7B/13B), Llama-3-8B, Phi-3, and Gemma2-2B.
4. Key Insights and Results
Complexity as a Feature
A key finding is that "smarter" model-generated data isn't always necessary to improve performance. By simply making the training task harder—for example, requiring a model to answer three instructions in reverse order—the model is forced to develop a more robust understanding of context and instruction dependencies.
Alleviating Memorization and Improving Efficiency
In standard instruction tuning, models often overfit, showing "stair-like" loss patterns as they memorize samples. Mosaic-IT's random shuffling ensures the model rarely sees the same overall sample twice, leading to smoother loss curves. Furthermore, packing up to 10 instructions into a single sample slashes training iterations, resulting in an 80% reduction in training time. On four NVIDIA A100 GPUs, training time dropped from 827 minutes to roughly 173 minutes for the same dataset.
Performance Gains
Mosaic-IT consistently outperformed baseline models. On Mistral-7B trained on Alpaca-GPT4, it achieved a 1.349 Pair-wise Winning Score. Massive improvements were seen on IF Eval, indicating superior constraint-following capabilities. In blind human evaluations, the Mosaic-IT model was preferred over the baseline 68% of the time on the Alpaca-GPT4 dataset.
5. Conclusion
Mosaic-IT provides a cost-free, effective method for enhancing LLM instruction following by synthesizing complexity from existing data. While currently focused on single-round scenarios, future work will explore multi-turn dialogues and more sophisticated semantic curation of mosaicked samples to further push the boundaries of model controllability.
Summary updated