Phased Instruction Fine-Tuning for Large Language Models

Wei Pang, Chuan Zhou, Xiao-Hua Zhou, Xiaojie Wang

Abstract

Instruction Fine-Tuning (IFT) enhances pre-trained language models from basic next-word prediction to complex instruction-following. However, existing One-off Instruction Fine-Tuning (One-off IFT) methods, applied to diverse instruction sets, may not effectively boost models' adherence to instructions due to the simultaneous handling of varying instruction complexities. To improve this, Phased Instruction Fine-Tuning (Phased IFT) is proposed, based on the idea that learning to follow instructions is a gradual process. It assesses instruction difficulty using GPT-4, divides the instruction data into subsets of increasing difficulty, and uptrains the model sequentially on these subsets. Experiments with Llama-2 7B/13B/70B, Llama-3 8B/70B, and Mistral-7B models using Alpaca data show that Phased IFT significantly outperforms One-off IFT, supporting the progressive alignment hypothesis and providing a simple and efficient way to enhance large language models.

Introduction

The transition from a base pre-trained model to a functional AI assistant typically relies on Instruction Fine-Tuning (IFT). In standard practice, this is a "one-off" process where a model is exposed to a diverse dataset—such as the 52K instructions in the Alpaca dataset—in a single training run. However, these datasets are inherently heterogeneous, containing a mix of simple linguistic tasks and highly complex reasoning or coding problems.

This research introduces the Progressive Alignment Hypothesis, suggesting that a model's ability to align with human intent is best developed through a gradual curriculum rather than simultaneous exposure to all complexity levels. To test this, the authors propose Phased Instruction Fine-Tuning (Phased IFT), a method that segments training into stages of increasing difficulty, effectively teaching the model to "walk before it runs."

Methodology

The Phased IFT pipeline consists of three primary stages designed to transform a raw instruction dataset into a structured curriculum:

1. Instruction Difficulty Calculating (IDC)

To quantify task complexity, the researchers utilize GPT-4 as an automated evaluator. Each instruction-input-output triplet in the dataset is assigned a score from 1 to 5. This score accounts for both the intrinsic difficulty of the prompt and the complexity required to generate a high-quality response.

2. Instruction Dataset Segmenting (IDS)

Once scored, the dataset is analyzed via a cumulative probability density curve. The researchers apply heuristic thresholds (1.5 and 3.5) to categorize the data into three distinct tiers:

  • Stage 1 (Easy): Foundational tasks (e.g., simple definitions or basic formatting).
  • Stage 2 (Medium): Tasks requiring moderate reasoning or multi-step instructions.
  • Stage 3 (Hard): Complex tasks involving coding, mathematical proofs, or nuanced creative writing.

3. Sequential Uptraining

Unlike standard IFT, which shuffles all data together, Phased IFT trains the model sequentially. The model first learns from Stage 1 data. The resulting checkpoint is then "uptrained" on Stage 2, and finally on Stage 3. This ensures the model masters foundational instruction-following before attempting high-complexity alignment.

Experimental Results

The efficacy of Phased IFT was evaluated across a wide range of model architectures, including Llama-2 (7B, 13B, 70B), Llama-3 (8B, 70B), and Mistral-7B. Performance was measured using GPT-4 as a judge across six benchmarks: Self-Instruct, WizardLM, Koala, Vicuna, OASST, and Anthropic.

Performance Gains

Phased IFT consistently outperformed the One-off IFT baseline across all model sizes:

  • Llama-2 7B: Achieved a +7.26% average win rate increase over the baseline.
  • Llama-3 8B: Showed a +7.64% improvement.
  • Llama-2 70B: Even at the largest scale, the phased approach yielded a +2.82% gain.

The Role of the Curriculum

A critical finding emerged when comparing Phased IFT to "Random Phased IFT" (where data is segmented into stages but shuffled randomly). The random approach actually performed worse than the one-off baseline. This confirms that the performance boost is not merely a result of multi-stage training or longer exposure, but specifically the order of difficulty.

Discussion and Insights

Dataset Composition

The scoring process revealed that the Alpaca dataset is heavily skewed toward simpler tasks, with 63.5% of instructions falling into the "Easy" category (score < 1.5). In a one-off training scenario, these simple examples may dilute the learning signal for more complex tasks. Phased IFT mitigates this by allowing the model to focus its capacity on high-difficulty samples only after the foundational behavior is established.

Redundancy and Interference

Ablation studies indicated that mixing easy data into later stages (re-training on Stage 1 and Stage 3 simultaneously) was less effective than pure sequential training. This suggests that once a model has reached a certain level of alignment, focusing exclusively on the next level of difficulty prevents the "interference" that occurs when simple and complex tasks are processed together.

Conclusion

Phased Instruction Fine-Tuning offers a simple yet powerful modification to the standard LLM alignment pipeline. By respecting the progressive nature of learning, Phased IFT enables models to achieve higher instruction-following performance without requiring additional data. Future work will focus on automating the selection of difficulty thresholds and exploring the optimal balance of sample sizes across different training stages.

Summary updated

Abstract

Instruction Fine-Tuning enhances pre-trained language models from basic next-word prediction to complex instruction-following. However, existing One-off Instruction Fine-Tuning (One-off IFT) method, applied on a diverse instruction, may not effectively boost models' adherence to instructions due to the simultaneous handling of varying instruction complexities. To improve this, Phased Instruction Fine-Tuning (Phased IFT) is proposed, based on the idea that learning to follow instructions is a gradual process. It assesses instruction difficulty using GPT-4, divides the instruction data into subsets of increasing difficulty, and uptrains the model sequentially on these subsets. Experiments with Llama-2 7B/13B/70B, Llama3 8/70B and Mistral-7B models using Alpaca data show that Phased IFT significantly outperforms One-off IFT, supporting the progressive alignment hypothesis and providing a simple and efficient way to enhance large language models. Codes and datasets from our experiments are freely available at https://github.com/xubuvd/PhasedSFT.
BibTeX
@inproceedings{pang2024phased,
  author = {Wei Pang and Chuan Zhou and Xiao-Hua Zhou and Xiaojie Wang},
  title = {{Phased Instruction Fine-Tuning for Large Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=Vr0XufbBkz},
}
Figure