Abstract
Fine-tuning large pre-trained language models with Evol-Instruct has achieved encouraging results across a wide range of tasks. However, designing effective evolving methods for instruction evolution requires substantial human expertise. This paper proposes Auto Evol-Instruct, an end-to-end framework that evolves instruction datasets using large language models without any human effort. The framework automatically analyzes and summarizes suitable evolutionary strategies for the given instruction data and iteratively improves the evolving method based on issues exposed during the instruction evolution process. Our extensive experiments demonstrate that the best method optimized by Auto Evol-Instruct outperforms human-designed methods on various benchmarks, including MT-Bench, AlpacaEval, GSM8K, and HumanEval.
Introduction
Instruction tuning is the essential process of fine-tuning Large Language Models (LLMs) on specific instruction-response pairs to ensure they are helpful and aligned with human intent. While the previous state-of-the-art approach, Evol-Instruct, successfully used LLMs to "evolve" simple instructions into more complex ones, it relied heavily on human-engineered heuristics. These manual rules—such as "add a constraint" or "increase reasoning depth"—are expensive to design and often fail to generalize across different domains like creative writing and mathematical proofs.
This paper introduces Auto Evol-Instruct, a framework that removes the human from the loop by allowing LLMs to analyze their own performance and refine their evolution strategies autonomously.
Methodology: The Auto Evol-Instruct Framework
The core of the framework is an iterative loop between two roles: the Evol LLM (the worker) and the Optimizer LLM (the manager).
The process follows four distinct stages:
- Initial Evolving Method: The process begins with a universal brainstorming prompt where the Evol LLM analyzes an instruction, lists potential methods for increasing complexity, and executes a plan.
- Evol Trajectory Analysis: The Optimizer LLM reviews the "evolution trajectory" to identify failures, such as Stagnant Complexity (where the model fails to increase difficulty) or Loss of Key Information.
- Evolving Method Optimization: Based on these identified failures, the Optimizer LLM rewrites the evolution instructions. For instance, it might add specific constraints to ensure the rewritten instruction remains understandable while increasing complexity.
- Multiple Optimizations: To ensure stability, the framework generates several potential methods in parallel and selects the one with the lowest failure rate on a development set.
Experimental Setup
The researchers tested the framework using several seed datasets: ShareGPT (10K samples), GSM8K Train (7K samples), and Code Alpaca (20K samples). These datasets were used to fine-tune various open-source models, including Mistral-7B, Mixtral-8x7B, and DeepSeek-Coder-33B. Evaluation was conducted on industry-standard benchmarks: MT-Bench and AlpacaEval for general chat, GSM8K for math, and HumanEval for coding.
Key Results and Insights
Auto Evol-Instruct consistently outperformed human-designed methods across all tested benchmarks. The framework demonstrated that LLMs are often better at identifying why an evolution failed than humans are at predicting how it might fail.
Key highlights include:
- General Chat: A Mixtral-8x7B model trained on 10K evolved samples achieved an 8.09 on MT-Bench, matching the performance of Claude 2.0.
- Mathematical Reasoning: On GSM8K, the framework boosted Mixtral-8x7B to 82.49% accuracy, surpassing specialized models like WizardMath-70B.
- Coding Proficiency: Using DeepSeek-Coder-33B, the method achieved 77.4% on HumanEval, outperforming GPT-3.5-Turbo.
- Optimization Dynamics: The study found a "sweet spot" for optimization; performance typically increases through approximately 12 steps before plateauing, suggesting that over-optimization can lead to superfluous complexity.
Conclusion and Future Work
Auto Evol-Instruct represents a significant step toward fully automated model alignment. By leveraging LLMs as prompt engineers, the framework produces higher-quality, more complex data than human-designed heuristics. Future research will focus on expanding the benchmark scope to broader knowledge tasks and investigating whether smaller, open-source models can effectively serve as optimizers for their own evolution strategies.
Summary updated