Abstract
Instruction Fine-Tuning (IFT) enhances pre-trained language models from basic next-word prediction to complex instruction-following. However, existing One-off Instruction Fine-Tuning (One-off IFT) methods, applied to diverse instruction sets, may not effectively boost models' adherence to instructions due to the simultaneous handling of varying instruction complexities. To improve this, Phased Instruction Fine-Tuning (Phased IFT) is proposed, based on the idea that learning to follow instructions is a gradual process. It assesses instruction difficulty using GPT-4, divides the instruction data into subsets of increasing difficulty, and uptrains the model sequentially on these subsets. Experiments with Llama-2 7B/13B/70B, Llama-3 8B/70B, and Mistral-7B models using Alpaca data show that Phased IFT significantly outperforms One-off IFT, supporting the progressive alignment hypothesis and providing a simple and efficient way to enhance large language models.
Introduction
The transition from a base pre-trained model to a functional AI assistant typically relies on Instruction Fine-Tuning (IFT). In standard practice, this is a "one-off" process where a model is exposed to a diverse dataset—such as the 52K instructions in the Alpaca dataset—in a single training run. However, these datasets are inherently heterogeneous, containing a mix of simple linguistic tasks and highly complex reasoning or coding problems.
This research introduces the Progressive Alignment Hypothesis, suggesting that a model's ability to align with human intent is best developed through a gradual curriculum rather than simultaneous exposure to all complexity levels. To test this, the authors propose Phased Instruction Fine-Tuning (Phased IFT), a method that segments training into stages of increasing difficulty, effectively teaching the model to "walk before it runs."
Methodology
The Phased IFT pipeline consists of three primary stages designed to transform a raw instruction dataset into a structured curriculum:
1. Instruction Difficulty Calculating (IDC)
To quantify task complexity, the researchers utilize GPT-4 as an automated evaluator. Each instruction-input-output triplet in the dataset is assigned a score from 1 to 5. This score accounts for both the intrinsic difficulty of the prompt and the complexity required to generate a high-quality response.
2. Instruction Dataset Segmenting (IDS)
Once scored, the dataset is analyzed via a cumulative probability density curve. The researchers apply heuristic thresholds (1.5 and 3.5) to categorize the data into three distinct tiers:
- Stage 1 (Easy): Foundational tasks (e.g., simple definitions or basic formatting).
- Stage 2 (Medium): Tasks requiring moderate reasoning or multi-step instructions.
- Stage 3 (Hard): Complex tasks involving coding, mathematical proofs, or nuanced creative writing.
3. Sequential Uptraining
Unlike standard IFT, which shuffles all data together, Phased IFT trains the model sequentially. The model first learns from Stage 1 data. The resulting checkpoint is then "uptrained" on Stage 2, and finally on Stage 3. This ensures the model masters foundational instruction-following before attempting high-complexity alignment.
Experimental Results
The efficacy of Phased IFT was evaluated across a wide range of model architectures, including Llama-2 (7B, 13B, 70B), Llama-3 (8B, 70B), and Mistral-7B. Performance was measured using GPT-4 as a judge across six benchmarks: Self-Instruct, WizardLM, Koala, Vicuna, OASST, and Anthropic.
Performance Gains
Phased IFT consistently outperformed the One-off IFT baseline across all model sizes:
- Llama-2 7B: Achieved a +7.26% average win rate increase over the baseline.
- Llama-3 8B: Showed a +7.64% improvement.
- Llama-2 70B: Even at the largest scale, the phased approach yielded a +2.82% gain.
The Role of the Curriculum
A critical finding emerged when comparing Phased IFT to "Random Phased IFT" (where data is segmented into stages but shuffled randomly). The random approach actually performed worse than the one-off baseline. This confirms that the performance boost is not merely a result of multi-stage training or longer exposure, but specifically the order of difficulty.
Discussion and Insights
Dataset Composition
The scoring process revealed that the Alpaca dataset is heavily skewed toward simpler tasks, with 63.5% of instructions falling into the "Easy" category (score < 1.5). In a one-off training scenario, these simple examples may dilute the learning signal for more complex tasks. Phased IFT mitigates this by allowing the model to focus its capacity on high-difficulty samples only after the foundational behavior is established.
Redundancy and Interference
Ablation studies indicated that mixing easy data into later stages (re-training on Stage 1 and Stage 3 simultaneously) was less effective than pure sequential training. This suggests that once a model has reached a certain level of alignment, focusing exclusively on the next level of difficulty prevents the "interference" that occurs when simple and complex tasks are processed together.
Conclusion
Phased Instruction Fine-Tuning offers a simple yet powerful modification to the standard LLM alignment pipeline. By respecting the progressive nature of learning, Phased IFT enables models to achieve higher instruction-following performance without requiring additional data. Future work will focus on automating the selection of difficulty thresholds and exploring the optimal balance of sample sizes across different training stages.
Summary updated