GRAFT: Execution-Guided Genetic Recombination for Agent Design

Sevda Ebrahimi, Mohammed Saidul Islam, Xiaodan Zhu, Elham DolatabadiarXiv 2026 · arXiv

TL;DR

GRAFT is a new evolutionary framework for automated LLM agent design that uses detailed execution feedback—not just final scores—to guide how agent architectures are combined and improved. By showing the "optimizer" LLM exactly where parent agents failed, it can perform targeted "semantic crossover," leading to a nearly 10% boost in accuracy over state-of-the-art baselines.

What Problem Does This Paper Address?

Building a high-performing LLM agent isn't just about the model; it’s about the architecture—the Python "scaffold" that handles task decomposition, tool use, and output verification. Manually designing these architectures is tedious and often relies on trial and error.

Automated search methods have emerged to solve this, often using evolutionary algorithms. In these systems, a population of agent designs "evolves" over generations. However, most existing methods are guided by aggregate fitness scores (e.g., "this agent got 70% correct"). The problem is that a single number hides why an agent failed. Two agents might both score 50%, but one might fail due to bad formatting while the other fails due to a logical error. Standard "blind" crossover operators can't distinguish between these cases, making the search process inefficient and less effective.

Methods

The authors introduce GRAFT (Execution-Guided Genetic Recombination for Agent Design). It follows a standard Genetic Algorithm (GA) loop but adds a crucial "Execution-Feedback Harness."

Figure 1: (a) The GA loop shared by GA-only search and GRAFT. GRAFT also passes execution samples from the evaluation step to the crossover step. (b) An illustrative crossover on Game24. Both parents score 50%, but their execution samples show different failures. Both children inherit Parent B’s check that the value equals 24, and each adds one step that neither parent has (new; integrated mutation). Without samples, the new step and combination are untargeted, so the answer 6 × 4 for [1, 3, 4, 6] still passes. With samples, the child adds a check that each number is used once, which rejects it.
  1. Execution-Feedback Harness: During the evaluation phase, GRAFT doesn't just record a score. It saves a bounded sample of successful and failed execution records (inputs, agent responses, and error messages).
  2. Semantic Crossover: When selecting two "parent" agents to create an "offspring," GRAFT provides the "optimizer" LLM with the parents' code, their fitness scores, and these specific execution samples.
  3. Integrated Mutation: Because the optimizer LLM can see exactly where a parent tripped up (e.g., a specific TypeError), it can perform "integrated mutation"—writing new code specifically designed to fix observed failures while merging the best features of both parents.

Crucially, this feedback is "free" in terms of compute; it reuses data already generated during the fitness evaluation, requiring no additional LLM calls to generate critiques or reflections.

Datasets

The researchers evaluated GRAFT across six diverse benchmarks spanning reasoning and programming:

  • ARC: Abstract reasoning (visual grid transformations).
  • MGSM: Multilingual grade-school math.
  • Game of 24: Arithmetic puzzles.
  • MBPP+: Python code generation.
  • SuperGPQA (STEM & Hard-Calc): Challenging graduate-level science and calculation questions.
  • CRUXEval-I: Reasoning about code execution.

Key Insights and Ideas

  • Aggregate scores are "lossy": The core "aha moment" is that instance-level failures are far more actionable than aggregate scores. Providing the LLM with a verbatim error message allows it to perform "task-specific interface adaptation"—essentially debugging the agent's architecture on the fly.
  • Qualitative Gradients: Execution traces act like qualitative gradients. Instead of just knowing "this is bad," the optimizer knows "this is bad because it returned a tuple instead of a single value."
  • Efficiency through Reuse: By harvesting traces from the evaluation step, GRAFT provides high-quality guidance without the massive overhead of "self-reflection" loops seen in other agent-tuning frameworks.
A case study on CRUXEval-I showing how GRAFT identifies a specific TypeError in parent runs and fixes it in the next generation.

Key Results

GRAFT significantly outperformed both hand-designed agents and existing automated design methods like ADAS and AFlow.

  • Overall Performance: Achieved a mean test accuracy of 82.0%, exceeding the strongest baseline by 9.8 percentage points.
  • Massive Gains in Hard Tasks: On the SuperGPQA-Hard-Calc benchmark, GRAFT reached 96.5% accuracy, compared to just 67.2% for the best baseline.
  • Ablation Proof: In a controlled test on the MGSM math task, adding the execution harness boosted accuracy from 56.7% to 91.6%, proving that the feedback—not just the evolutionary search—is the primary driver of success.
Figure 3: Effect of the behavioral-feedback harness. Test accuracy (%) of the best-by-validation agent found by genetic search alone (GA-only) and by GRAFT (genetic search with the harness) on MGSM and ARC, Qwen3.5-35B-A3B as both optimizer and executor.

Limitations and Future Work

  • Task Scope: The evaluation focused on reasoning and coding. It remains to be seen how GRAFT performs in open-ended, interactive, or long-horizon environments (like web navigation).
  • Model Specificity: An architecture discovered using one "executor" model (e.g., Qwen) might not transfer perfectly to another (e.g., Llama) without further optimization.
  • Distribution Shift: The study doesn't establish how robust these discovered architectures are when faced with entirely new types of data outside the evaluation set.
  • Future Directions: The authors suggest exploring how discovered architectures might transfer across tasks and scaling the search to even more complex agentic systems.

Summary updated

Abstract

Automated agent design searches for executable architectures that improve LLM-based agent performance, with evolutionary methods using language-model crossover to recombine promising mechanisms for task decomposition, model coordination, and verification. This search, however, is typically guided by aggregate fitness alone, which hides the instance-level successes and failures behind each score. We introduce Execution Guided Genetic Recombination for Agent Design (GRAFT), which uses aggregate fitness for selection and survival and parent-specific execution evidence to guide semantic crossover with integrated mutation. Its feedback harness reuses bounded samples of successful and failed runs from fitness evaluation, withholding reference answers and requiring no additional task executions. We evaluate GRAFT across six reasoning and programming settings using open-weight optimizer and executor models. It achieves a mean test accuracy of 82.0%, exceeding the strongest baseline by 9.8% points. Controlled ablations show that adding execution evidence improves test accuracy with initialization, search rules, repair policies, and evaluation budgets held fixed. Our results support the value of parent-specific execution evidence for evolutionary agent design.
BibTeX
@misc{ebrahimi2026graft,
  author = {Sevda Ebrahimi and Mohammed Saidul Islam and Xiaodan Zhu and Elham Dolatabadi},
  title = {{GRAFT: Execution-Guided Genetic Recombination for Agent Design}},
  year = {2026},
  eprint = {2611.23509},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23509},
}
Figure