Instruction Tuning with GPT-4

Baolin Peng, Chunyuan Li, Pengcheng He, Michel Galley, Jianfeng GaoarXiv 2023 · OpenReview

TL;DR

This paper introduces the first use of GPT-4 to generate instruction-following data for finetuning large language models (LLMs). The resulting instruction-tuned LLaMA models demonstrate superior zero-shot performance compared to models trained on data generated by previous state-of-the-art models, highlighting GPT-4's potential for creating high-quality training data.

What Problem Does This Paper Address?

Large Language Models (LLMs) have shown impressive capabilities like in-context learning and chain-of-thought reasoning. However, to make them truly useful for real-world tasks, they need to understand and follow natural language instructions. Instruction tuning is a key technique for achieving this, typically involving finetuning LLMs on datasets of instructions and their corresponding outputs. While human-annotated data can be effective, it's expensive and time-consuming. Previous work has explored using existing LLMs (like GPT-3.5) to generate this data automatically, a method known as "Self-Instruct." This paper addresses the challenge of creating even higher-quality instruction-following data to further improve the capabilities of open-source LLMs, leveraging the advanced reasoning and generation abilities of the newly released GPT-4.

Methods

The core of this paper is the use of GPT-4 as a "teacher" model to generate instruction-following data. The authors adapted the Self-Instruct methodology, using GPT-4 to create instruction-output pairs. They generated 52K English and Chinese instruction-following datasets. Crucially, they also collected feedback and comparison data from GPT-4 itself, which was used to train reward models. This reward modeling component is essential for evaluating and further aligning LLMs, especially in the context of Reinforcement Learning from Human Feedback (RLHF), though RLHF itself was not fully implemented in this work. The authors then finetuned LLaMA models using this GPT-4 generated data.

Datasets

  • Alpaca Dataset: The paper reuses 52K unique instructions from the Alpaca dataset as the basis for their instruction-following data.
  • GPT-4 Generated English Instruction-Following Data: 52K English instruction-output pairs generated by GPT-4.
  • GPT-4 Generated Chinese Instruction-Following Data: 52K Chinese instruction-output pairs, created by translating Alpaca instructions and having GPT-4 answer them in Chinese.
  • GPT-4 Generated Feedback and Comparison Data: Data where GPT-4 rates its own responses and compares them against other models (GPT-3.5, OPT-IML).
  • Unnatural Instructions Dataset: A dataset of 68K instruction-input-output triplets used for evaluating the gap between GPT-4 and their finetuned models.

Key Insights and Ideas

  • GPT-4 as a Superior Data Generator: The most significant insight is that GPT-4, when used for instruction data generation, produces outputs that lead to demonstrably better zero-shot performance in finetuned LLMs compared to data generated by previous models like GPT-3.5.
  • Cross-Lingual Transfer Potential: The creation of a Chinese instruction-following dataset and its use in finetuning LLaMA suggests promising avenues for cross-language generalization in LLMs.
  • GPT-4 for Reward Modeling: Leveraging GPT-4's ability to provide nuanced feedback and comparisons is a cost-effective way to generate data for training reward models, which is a crucial step towards more advanced LLM alignment.
  • Data Quality Over Quantity (to an extent): While the dataset size (52K) is comparable to Alpaca, the quality of GPT-4 generated data appears to be the driving factor behind performance improvements.

Key Results

  • Improved Zero-Shot Performance: Instruction-tuned LLaMA models using GPT-4 generated data achieved superior zero-shot performance on new tasks compared to models trained on GPT-3.5 generated data.
  • Human Evaluation: In human evaluations using the HHH (Helpfulness, Honesty, Harmlessness) criteria, LLaMA finetuned with GPT-4 data performed comparably to the original GPT-4 model, significantly outperforming LLaMA finetuned with GPT-3 data (Alpaca).
  • Automatic Evaluation: Automatic evaluations using GPT-4 as a judge showed that LLaMA finetuned with GPT-4 data achieved higher scores than Alpaca and even outperformed the baseline LLaMA model.
  • Response Length: GPT-4 tends to generate longer responses than GPT-3.5.

Limitations and Future Work

  • Data Scale: The authors acknowledge that while their 52K dataset is effective, larger datasets (like Vicuna's 700K) and larger base models (like LLaMA 13B) could lead to even higher performance. They plan to continue collecting GPT-4 data and combining it with other sources.
  • RLHF Implementation: The reward models were primarily used for evaluation and ranking, not for full RLHF training. The paper suggests that training LLMs with these reward models for reinforcement learning is a natural next step.
  • Model Size: The experiments were conducted on LLaMA 7B models. Exploring larger LLaMA models with GPT-4 generated data is a clear direction for future work.
  • Unnatural Instructions: While LLaMA-GPT4 showed promise on creative tasks, its performance on "unnatural instructions" (Figure 6) was noted to be slightly lower than Alpaca in some length categories, suggesting room for improvement in handling more complex or creative prompts.

Summary updated

Abstract

Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available.
BibTeX
@misc{peng2023instruction,
  author = {Baolin Peng and Chunyuan Li and Pengcheng He and Michel Galley and Jianfeng Gao},
  title = {{Instruction Tuning with GPT-4}},
  year = {2023},
  eprint = {2304.03277},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2304.03277},
}
Figure