TL;DR
By batching multiple mutation proposals in parallel, the GEPA optimization loop for agentic AI systems can achieve a 3–4× wall-clock speedup while simultaneously improving held-out test scores. This approach transforms a traditionally sequential, resource-heavy bottleneck into a parallelizable process that generalizes better to new data.
What Problem Does This Paper Address?
Optimizing an "agentic" AI system—improving the prompts or code that guide an LLM—is a notoriously slow and expensive process. State-of-the-art optimizers like GEPA (Generalized Evolutionary Prompt Adaptation) use a "reflective" loop: the system proposes a mutation, evaluates it, and uses that feedback to inform the next step.
The problem is that this loop is inherently sequential. Even if you have a massive cluster of GPUs or a high-rate API tier, the optimizer typically waits for one candidate to be fully evaluated before proposing the next. This leaves parallel evaluation capacity idle and can result in optimization runs that take many hours. Furthermore, repeatedly "steering" an agent based on a single small validation set can lead to overfitting, where the agent performs great on the validation data but fails on the actual test set.
Methods
The authors introduce Parallel Proposals, a $P \times N$ sampling strategy that extends the open-source GEPA framework.
Instead of the standard sequential approach (sampling one parent and one mutation), the batched iteration works as follows:
- Sample $P$ parents from the current "Pareto frontier" (the set of best-performing candidates across different trade-offs).
- Generate $N$ mutations for each parent using different mini-batches of data.
- Evaluate all $P \cdot N$ candidates concurrently.
By dispatching these reflection requests and evaluations in one batch, the system pays the "iteration latency" once for multiple candidates. The authors also provide a runtime model to predict speedups based on the size of the worker pool and the probability of a mutation being successful.
Datasets
The researchers evaluated their approach on two distinct benchmarks:
- LiveBench-Math: A challenging set of competition-level math problems (AMC, AIME, etc.). The goal was to optimize a solver prompt with a budget of 5,000 metric calls.
- HoVer: A multi-hop claim verification task requiring the agent to retrieve specific Wikipedia pages. The goal was to optimize a retrieval agent with a budget of 3,000 metric calls.
Key Insights and Ideas
- Parallelism as Regularization: One of the most surprising findings is that batching doesn't just make things faster; it makes them better. By committing to multiple proposals simultaneously before adapting to validation results, the optimizer is less likely to overfit to the specific quirks of the validation set.
- The "Faster and Better" Frontier: In many optimization problems, there is a trade-off between speed and quality. Here, parallel proposals move the needle in both directions at once, as seen in the primary results plot.
- Task-Dependent Scaling: The optimal way to split the "width" of a batch depends on the task. For math problems (LiveBench), generating multiple mutations for a single parent ($N$) was more effective. For retrieval tasks (HoVer), spreading mutations across more parents ($P$) yielded better results.
- Diminishing Returns of Width: While parallelism is great, it's limited by the "slowest" call in a batch and the total size of the worker pool. The authors' model accurately predicts where these diminishing returns kick in.
Key Results
The batched approach outperformed the sequential baseline across the board:
- Speed: Most batched runs finished in half the wall-clock time or less. The fastest configurations achieved a 3× to 4× speedup.
- Performance:
- On LiveBench-Math, scores rose from 68.9% to 72.1% (using a $2 \times 2$ configuration).
- On HoVer, scores jumped significantly from 49.0% to 60.0% (using an $8 \times 1$ configuration).
- Efficiency: Batched settings reached high validation scores much earlier in their "dollar budget" than sequential runs.
Limitations and Future Work
The authors identify that while they have successfully scaled the "width" of the optimization step ($P$ and $N$), there are other axes of scaling yet to be fully explored. These include the size of the mini-batch used for reflection and the amount of compute (e.g., chain-of-thought) spent on generating each mutation.
A major open question remains: given a fixed budget of dollars or API calls, how should a user automatically allocate that budget across these different "dials" (parents, mutations, mini-batch size, and reflection compute) to get the best possible agent? This work provides the first look at those trade-offs but leaves the automated allocation for future research.
Summary updated