Automatic Instruction Evolving for Large Language Models

Weihao Zeng, Can Xu, Yingxiu Zhao, Jian-Guang Lou, Weizhu Chen

Abstract

Fine-tuning large pre-trained language models with Evol-Instruct has achieved encouraging results across a wide range of tasks. However, designing effective evolving methods for instruction evolution requires substantial human expertise. This paper proposes Auto Evol-Instruct, an end-to-end framework that evolves instruction datasets using large language models without any human effort. The framework automatically analyzes and summarizes suitable evolutionary strategies for the given instruction data and iteratively improves the evolving method based on issues exposed during the instruction evolution process. Our extensive experiments demonstrate that the best method optimized by Auto Evol-Instruct outperforms human-designed methods on various benchmarks, including MT-Bench, AlpacaEval, GSM8K, and HumanEval.

Introduction

Instruction tuning is the essential process of fine-tuning Large Language Models (LLMs) on specific instruction-response pairs to ensure they are helpful and aligned with human intent. While the previous state-of-the-art approach, Evol-Instruct, successfully used LLMs to "evolve" simple instructions into more complex ones, it relied heavily on human-engineered heuristics. These manual rules—such as "add a constraint" or "increase reasoning depth"—are expensive to design and often fail to generalize across different domains like creative writing and mathematical proofs.

This paper introduces Auto Evol-Instruct, a framework that removes the human from the loop by allowing LLMs to analyze their own performance and refine their evolution strategies autonomously.

Methodology: The Auto Evol-Instruct Framework

The core of the framework is an iterative loop between two roles: the Evol LLM (the worker) and the Optimizer LLM (the manager).

Overview of the Auto Evol-Instruct framework, illustrating the iterative loop between the Evol LLM and the Optimizer LLM to refine instruction evolution.

The process follows four distinct stages:

  1. Initial Evolving Method: The process begins with a universal brainstorming prompt where the Evol LLM analyzes an instruction, lists potential methods for increasing complexity, and executes a plan.
  2. Evol Trajectory Analysis: The Optimizer LLM reviews the "evolution trajectory" to identify failures, such as Stagnant Complexity (where the model fails to increase difficulty) or Loss of Key Information.
  3. Evolving Method Optimization: Based on these identified failures, the Optimizer LLM rewrites the evolution instructions. For instance, it might add specific constraints to ensure the rewritten instruction remains understandable while increasing complexity.
  4. Multiple Optimizations: To ensure stability, the framework generates several potential methods in parallel and selects the one with the lowest failure rate on a development set.

Experimental Setup

The researchers tested the framework using several seed datasets: ShareGPT (10K samples), GSM8K Train (7K samples), and Code Alpaca (20K samples). These datasets were used to fine-tune various open-source models, including Mistral-7B, Mixtral-8x7B, and DeepSeek-Coder-33B. Evaluation was conducted on industry-standard benchmarks: MT-Bench and AlpacaEval for general chat, GSM8K for math, and HumanEval for coding.

Key Results and Insights

Auto Evol-Instruct consistently outperformed human-designed methods across all tested benchmarks. The framework demonstrated that LLMs are often better at identifying why an evolution failed than humans are at predicting how it might fail.

Performance comparison across MT-Bench, GSM8K, and HumanEval, showing Auto Evol-Instruct outperforming human-designed methods.

Key highlights include:

  • General Chat: A Mixtral-8x7B model trained on 10K evolved samples achieved an 8.09 on MT-Bench, matching the performance of Claude 2.0.
  • Mathematical Reasoning: On GSM8K, the framework boosted Mixtral-8x7B to 82.49% accuracy, surpassing specialized models like WizardMath-70B.
  • Coding Proficiency: Using DeepSeek-Coder-33B, the method achieved 77.4% on HumanEval, outperforming GPT-3.5-Turbo.
  • Optimization Dynamics: The study found a "sweet spot" for optimization; performance typically increases through approximately 12 steps before plateauing, suggesting that over-optimization can lead to superfluous complexity.

Conclusion and Future Work

Auto Evol-Instruct represents a significant step toward fully automated model alignment. By leveraging LLMs as prompt engineers, the framework produces higher-quality, more complex data than human-designed heuristics. Future research will focus on expanding the benchmark scope to broader knowledge tasks and investigating whether smaller, open-source models can effectively serve as optimizers for their own evolution strategies.

Summary updated

Abstract

Fine-tuning large pre-trained language models with Evol-Instruct has achieved encouraging results across a wide range of tasks. However, designing effective evolving methods for instruction evolution requires substantial human expertise. This paper proposes Auto Evol-Instruct, an end-to-end framework that evolves instruction datasets using large language models without any human effort. The framework automatically analyzes and summarizes suitable evolutionary strategies for the given instruction data and iteratively improves the evolving method based on issues exposed during the instruction evolution process. Our extensive experiments demonstrate that the best method optimized by Auto Evol-Instruct outperforms human-designed methods on various benchmarks, including MT-Bench, AlpacaEval, GSM8K, and HumanEval.
BibTeX
@inproceedings{zeng2024automatic,
  author = {Weihao Zeng and Can Xu and Yingxiu Zhao and Jian-Guang Lou and Weizhu Chen},
  title = {{Automatic Instruction Evolving for Large Language Models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  url = {https://openreview.net/forum?id=VlvlIU2Fyq},
}
Figure