Abstract
Fine-tuning large pre-trained language models with Evol-Instruct has achieved encouraging results across a wide range of tasks. However, designing effective evolving methods for instruction evolution requires substantial human expertise. This paper proposes Auto Evol-Instruct, an end-to-end framework that evolves instruction datasets using large language models without any human effort. The framework automatically analyzes and summarizes suitable evolutionary strategies for the given instruction data and iteratively improves the evolving method based on issues exposed during the instruction evolution process. Our extensive experiments demonstrate that the best method optimized by Auto Evol-Instruct outperforms human-designed methods on various benchmarks, including MT-Bench, AlpacaEval, GSM8K, and HumanEval.
Introduction
Instruction tuning has emerged as a critical phase in the development of Large Language Models (LLMs), transforming base models into capable assistants like ChatGPT. The efficacy of this process is heavily dependent on the quality, diversity, and complexity of the instruction data used. To address the limitations of static datasets, researchers previously introduced Evol-Instruct, a method that uses LLMs to rewrite simple instructions into more complex versions.
While effective, Evol-Instruct relies on human-engineered heuristics—such as "add a constraint" or "deepen the reasoning"—to guide the evolution process. These manual rules are not only labor-intensive to design but often fail to generalize across different domains; a strategy effective for creative writing may be ineffective for mathematical reasoning. This paper introduces Auto Evol-Instruct, a framework designed to automate the design of these evolution strategies, allowing LLMs to autonomously refine their own training data.
The Auto Evol-Instruct Framework
Auto Evol-Instruct treats the design of instruction-evolution strategies as an iterative optimization problem. The framework employs two primary components: an Evol LLM, which performs the actual rewriting of the data, and an Optimizer LLM (typically a high-capacity model like GPT-4), which critiques and improves the evolution instructions.
The Optimization Loop
The framework operates through a three-stage iterative process:
- Initial Evolving Method Design: The process begins with a "universal" prompt that encourages the LLM to brainstorm various ways to increase the complexity of a given instruction.
- Evol Trajectory Analysis: The Optimizer LLM examines the "trajectory" of the evolved instructions to identify specific failures. The authors categorize these failures into three primary types:
- Stagnant Complexity: The instruction was rewritten but did not actually become more difficult.
- Insufficient Qualification: The instruction became too vague or ambiguous to be answered correctly.
- Loss of Key Information: Essential constraints or data from the original prompt were dropped during the evolution.
- Evolving Method Optimization: Based on the identified failures, the Optimizer LLM rewrites the evolution prompt. For instance, if the model consistently loses track of variables in math problems, the optimizer might inject a requirement to "identify and preserve all numerical constants."
To ensure the stability of the generated strategies, the framework utilizes a "Multiple Optimizations" strategy, where several candidate methods are generated in parallel. The system then selects the method with the lowest failure rate on a small development subset.
Experimental Results
The authors validated Auto Evol-Instruct across three distinct domains: general instruction following (using ShareGPT), mathematical reasoning (GSM8K), and code generation (Code Alpaca). The resulting models were evaluated against standard benchmarks, consistently outperforming human-designed methods.
Performance Highlights
- General Chat: When applied to the Mixtral-8x7B model, Auto Evol-Instruct achieved a score of 8.09 on MT-Bench, surpassing both GPT-3.5-Turbo (7.90) and the original human-designed Evol-Instruct (7.76).
- Mathematics: On the GSM8K benchmark, the framework boosted Mistral-7B's performance by 13.84 points over the base seed data, reaching a final score of 70.74.
- Coding: Using evolved Code Alpaca data, a DeepSeek-Coder-33B model achieved 77.4% on HumanEval, outperforming both WizardCoder-34B and GPT-3.5-Turbo.
Discussion and Key Insights
The Meta-Evolution Shift
The core innovation of this work is the shift from evolving data to evolving the method used to create that data. By allowing the system to discover domain-specific strategies autonomously, it uncovers nuances—such as focusing on logical consistency in math—that human engineers might overlook.
Failure as a Learning Signal
By explicitly defining and detecting evolution failures, the system creates a feedback loop that suppresses common LLM tendencies, such as "hallucinated" complexity or redundant phrasing. This self-correction mechanism is vital for maintaining data integrity as complexity increases.
Efficiency and Scalability
The framework is computationally efficient, requiring only a small subset of data (~2,000 entries) to identify an optimal evolution strategy. Once the optimal prompt is found, it can be applied to the entire dataset. Furthermore, the "Mix Rounds" experiments indicate that data evolved through this automated process scales more effectively than manual methods, showing consistent performance gains as more evolved data is integrated.
Limitations
Despite its success, the framework currently depends on high-end proprietary models like GPT-4 to serve as the Optimizer LLM. Additionally, the authors noted a point of diminishing returns: "over-optimization" (typically beyond 12 iterations) can lead to the accumulation of superfluous information, which may eventually degrade the quality of the instruction data.
Conclusion
Auto Evol-Instruct represents a significant step toward fully autonomous model improvement. By removing the human bottleneck in data engineering, the framework allows for the creation of highly specialized, high-complexity datasets tailored to specific tasks. Future research will focus on whether smaller, open-source models can effectively serve as optimizers and expanding the method to a broader range of benchmarks.
Summary updated