TL;DR
GRAFT is a new evolutionary framework for automated LLM agent design that uses detailed execution feedback—not just final scores—to guide how agent architectures are combined and improved. By showing the "optimizer" LLM exactly where parent agents failed, it can perform targeted "semantic crossover," leading to a nearly 10% boost in accuracy over state-of-the-art baselines.
What Problem Does This Paper Address?
Building a high-performing LLM agent isn't just about the model; it’s about the architecture—the Python "scaffold" that handles task decomposition, tool use, and output verification. Manually designing these architectures is tedious and often relies on trial and error.
Automated search methods have emerged to solve this, often using evolutionary algorithms. In these systems, a population of agent designs "evolves" over generations. However, most existing methods are guided by aggregate fitness scores (e.g., "this agent got 70% correct"). The problem is that a single number hides why an agent failed. Two agents might both score 50%, but one might fail due to bad formatting while the other fails due to a logical error. Standard "blind" crossover operators can't distinguish between these cases, making the search process inefficient and less effective.
Methods
The authors introduce GRAFT (Execution-Guided Genetic Recombination for Agent Design). It follows a standard Genetic Algorithm (GA) loop but adds a crucial "Execution-Feedback Harness."
- Execution-Feedback Harness: During the evaluation phase, GRAFT doesn't just record a score. It saves a bounded sample of successful and failed execution records (inputs, agent responses, and error messages).
- Semantic Crossover: When selecting two "parent" agents to create an "offspring," GRAFT provides the "optimizer" LLM with the parents' code, their fitness scores, and these specific execution samples.
- Integrated Mutation: Because the optimizer LLM can see exactly where a parent tripped up (e.g., a specific
TypeError), it can perform "integrated mutation"—writing new code specifically designed to fix observed failures while merging the best features of both parents.
Crucially, this feedback is "free" in terms of compute; it reuses data already generated during the fitness evaluation, requiring no additional LLM calls to generate critiques or reflections.
Datasets
The researchers evaluated GRAFT across six diverse benchmarks spanning reasoning and programming:
- ARC: Abstract reasoning (visual grid transformations).
- MGSM: Multilingual grade-school math.
- Game of 24: Arithmetic puzzles.
- MBPP+: Python code generation.
- SuperGPQA (STEM & Hard-Calc): Challenging graduate-level science and calculation questions.
- CRUXEval-I: Reasoning about code execution.
Key Insights and Ideas
- Aggregate scores are "lossy": The core "aha moment" is that instance-level failures are far more actionable than aggregate scores. Providing the LLM with a verbatim error message allows it to perform "task-specific interface adaptation"—essentially debugging the agent's architecture on the fly.
- Qualitative Gradients: Execution traces act like qualitative gradients. Instead of just knowing "this is bad," the optimizer knows "this is bad because it returned a tuple instead of a single value."
- Efficiency through Reuse: By harvesting traces from the evaluation step, GRAFT provides high-quality guidance without the massive overhead of "self-reflection" loops seen in other agent-tuning frameworks.
Key Results
GRAFT significantly outperformed both hand-designed agents and existing automated design methods like ADAS and AFlow.
- Overall Performance: Achieved a mean test accuracy of 82.0%, exceeding the strongest baseline by 9.8 percentage points.
- Massive Gains in Hard Tasks: On the
SuperGPQA-Hard-Calcbenchmark, GRAFT reached 96.5% accuracy, compared to just 67.2% for the best baseline. - Ablation Proof: In a controlled test on the MGSM math task, adding the execution harness boosted accuracy from 56.7% to 91.6%, proving that the feedback—not just the evolutionary search—is the primary driver of success.
Limitations and Future Work
- Task Scope: The evaluation focused on reasoning and coding. It remains to be seen how GRAFT performs in open-ended, interactive, or long-horizon environments (like web navigation).
- Model Specificity: An architecture discovered using one "executor" model (e.g., Qwen) might not transfer perfectly to another (e.g., Llama) without further optimization.
- Distribution Shift: The study doesn't establish how robust these discovered architectures are when faced with entirely new types of data outside the evaluation set.
- Future Directions: The authors suggest exploring how discovered architectures might transfer across tasks and scaling the search to even more complex agentic systems.
Summary updated