Phased Instruction Fine-Tuning for Large Language Models

Wei Pang, Chuan Zhou, Xiao-Hua Zhou, Xiaojie Wang

Abstract

Instruction Fine-Tuning, a method enhancing pre-trained language models’ capabilities from mere next-word prediction to complex instruction following, often employs a one-off training approach on diverse instruction dataset. However, this method may not effectively enhance models’ adherence to instructions due to the simultaneous handling of varying instruction complexities. To address this, we propose a novel phased instruction fine-tuning (Phased IFT) method, grounded in the hypothesis of progressive alignment, which posits that the transition of a pre-trained language model from simple next-word prediction to sophisticated instruction following is a gradual learning process. Specifically, we obtain the score of difficulty for each instruction via GPT-4, stratify the instruction data into subsets of increasing difficulty, and sequentially uptrain on these subsets using the standard supervised loss. Through extensive experiments on the pre-trained models Llama-2 7B/13B, and Mistral-7B using the 52K Alpaca instruction data, we demonstrate that Phased IFT significantly surpasses traditional one-off instruction fine-tuning (One-off IFT) method in win rate, empirically validating the progressive alignment hypothesis. Our findings suggest that Phased IFT offers a simple yet effective pathway for elevating the instruction-following capabilities of pre-trained language models.

Introduction

Large Language Models (LLMs) are typically pre-trained on massive raw text corpora to predict the next word. To transform them into useful assistants, they undergo Instruction Fine-Tuning (IFT) to learn human intent. The conventional "One-off IFT" approach feeds a diverse dataset—ranging from simple arithmetic to complex reasoning—to the model simultaneously.

However, this approach ignores the nuanced differences in instruction difficulty. Much like a human student requires a curriculum that builds from basic to advanced concepts, an LLM may struggle to align its internal representations when presented with all levels of complexity at once. This paper introduces Phased IFT to address this inefficiency through a curriculum-based learning strategy.

Methods: The Phased IFT Framework

The Phased IFT method is a sequential training framework designed to facilitate gradual alignment. It consists of three primary components:

  1. Instruction Difficulty Calculating (IDC): GPT-4 is utilized as a teacher to score (instruction, input, output) triplets on a scale of 1 to 5, considering task complexity and response challenge.
  2. Instruction Dataset Segmenting (IDS): The dataset is divided into three difficulty-based stages: Stage 1 (Easy, scores [1, 1.5)), Stage 2 (Medium, scores [1.5, 3.5)), and Stage 3 (Hard, scores [3.5, 5.5]).
  3. Sequential Uptraining: The model is trained on Stage 1 first. The resulting checkpoint is then used as the starting point for Stage 2, and finally for Stage 3. Consistent hyperparameters are maintained across stages to isolate the impact of the data curriculum.
The Phased IFT framework: scoring instructions with GPT-4, segmenting by difficulty, and sequential uptraining.

Results and Discussion

The researchers evaluated Phased IFT using the Alpaca 52K and Alpaca-cleaned datasets across several model families, including Llama-2 (7B, 13B, 70B), Llama-3 (8B, 70B), and Mistral-7B. Performance was measured via win rates against a one-off baseline across six benchmarks (e.g., Vicuna, WizardLM, Anthropic).

Phased IFT consistently outperformed the baseline across almost all models and benchmarks. For instance, Llama-3 8B showed a +7.64% average win rate increase, while Llama-2 70B achieved a +7.59% boost on the Alpaca-cleaned dataset. The "easy-to-hard" sequence proved to be the dominant strategy, often yielding double-digit improvements on specific benchmarks.

Win rate improvement across sequential training stages for various Llama and Mistral models.

Key Insights

  • Progressive Alignment: The data supports the hypothesis that mastering simpler instructions first builds a foundation for complex task following. Training in reverse (hard-to-easy) often performed worse than the baseline.
  • Curriculum vs. Multi-Stage: Ablation studies using randomly segmented stages failed to replicate the gains of Phased IFT, confirming that the difficulty gradient—not just the multi-stage process—is the primary driver of performance.
  • High-Leverage Hard Data: Although Stage 3 contains the fewest samples, it contributes significantly to the final performance jump, highlighting the importance of high-difficulty data in the final alignment phase.

Conclusion

Phased IFT provides a simple yet effective pathway for elevating the instruction-following capabilities of LLMs. While effective, the method currently relies on manual thresholds for difficulty segmentation and proprietary models for scoring. Future research may focus on automating "phase boundaries" and exploring open-source alternatives for difficulty grading.

Summary updated

Abstract

Instruction Fine-Tuning, a method enhancing pre-trained language models’ capabilities from mere next-word prediction to complex instruction following, often employs a one-off training approach on diverse instruction dataset. However, this method may not effectively enhance models’ adherence to instructions due to the simultaneous handling of varying instruction complexities. To address this, we propose a novel phased instruction fine-tuning (Phased IFT) method, grounded in the hypothesis of progressive alignment, which posits that the transition of a pre-trained language model from simple next-word prediction to sophisticated instruction following is a gradual learning process. Specifically, we obtain the score of difficulty for each instruction via GPT-4, stratify the instruction data into subsets of increasing difficulty, and sequentially uptrain on these subsets using the standard supervised loss. Through extensive experiments on the pre-trained models Llama-2 7B/13B, and Mistral-7B using the 52K Alpaca instruction data, we demonstrate that Phased IFT significantly surpasses traditional one-off instruction fine-tuning (One-off IFT) method in win rate, empirically validating the progressive alignment hypothesis. Our findings suggest that Phased IFT offers a simple yet effective pathway for elevating the instruction-following capabilities of pre-trained language models.
BibTeX
@inproceedings{pang2024phased,
  author = {Wei Pang and Chuan Zhou and Xiao-Hua Zhou and Xiaojie Wang},
  title = {{Phased Instruction Fine-Tuning for Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=Vr0XufbBkz},
}
Figure