TL;DR
DuoDiscover is a new search system that scales autonomous engineering discovery by overcoming the "evaluation bottleneck"—the massive time cost of running full benchmarks. By having a fleet of agents co-discover fast proxy evaluators (called "yardsticks") alongside optimization targets, it achieves significant speedups in complex systems like vLLM and CPU microarchitecture, often outperforming both human experts and state-of-the-art AI baselines.
What Problem Does This Paper Address?
In the quest for autonomous AI discovery—where agents optimize code, algorithms, or hardware—the primary hurdle isn't just writing the code; it's knowing if the code is actually better.
In real-world engineering, a "true evaluation" (like measuring the throughput of a large-scale serving engine or simulating a CPU design) can take anywhere from 15 minutes to 28 hours. If your total search budget is 20 hours, you can only afford a handful of these evaluations. This makes traditional search methods, which rely on constant feedback, painfully slow and ineffective. The authors identify this as the evaluation bottleneck.
Methods
DuoDiscover tackles this bottleneck through three main mechanisms:
- Co-discovery of Targets and Yardsticks: Instead of evaluating every small code change against the full, expensive benchmark, each worker agent identifies a specific target (a function or module) and builds a yardstick—a fast, tailored proxy evaluator (e.g., a microbenchmark or a cost model). The agent then "hill-climbs" the yardstick in a fast inner loop (seconds) before sending promising candidates to the slow outer loop (hours) for true evaluation.
- Fidelity Testing: Proxy evaluators are prone to "reward hacking," where an agent finds a way to improve the proxy score without actually improving the true objective. DuoDiscover continually runs fidelity tests using Kendall’s $\tau$ rank correlation to ensure the yardstick's ranking still matches the true objective. If a yardstick drifts, it is invalidated.
- Bottom-up Fleet Orchestration: A lightweight orchestrator manages a fleet of independent agents. It doesn't follow a rigid plan; instead, it terminates stalled agents, briefs others on successful discoveries, and "seeds" new agents with confirmed improvements from their peers.
Datasets
The authors evaluated DuoDiscover across a diverse range of benchmarks spanning the entire computing stack:
- Repository-level Optimization: FormulaCode (scientific Python) and SWE-fficiency (popular libraries like NumPy/pandas).
- Kernel Suites: SOL-Exec (235 GPU kernels), JAXBench (TPU operators), TPU-Inference, and FA4 (Flash Attention 4).
- Real-world Systems: The vLLM serving engine, SQLite database, and zstd compression library.
- Hardware Design: The DPC4 (Data Prefetching Championship) CPU microarchitecture task.
Key Insights and Ideas
- Yardsticks are Target-Specific: Building a single fast proxy for an entire codebase is nearly impossible. However, building a fast proxy for a single function is much more feasible. DuoDiscover exploits this by letting agents define their own narrow scope.
- Fidelity is a Moving Target: A yardstick that is accurate at the start of a search might become inaccurate as the code evolves. The "aha moment" here is that validation must be continuous, not just a one-time setup.
- Search Efficiency over Token Volume: DuoDiscover often uses significantly fewer true evaluations and tokens than baselines like Claude Code to reach the same performance level because it spends its "thinking budget" on high-fidelity candidates.
- Bottom-up Coordination: By allowing agents to pick their own targets and then having the orchestrator combine their gains, the system avoids the need for a complex central planner that understands the entire system architecture.
Key Results
DuoDiscover consistently outperformed human experts and specialized AI baselines:
- vLLM: Achieved a 1.92× geometric mean gain across four service-level objectives (latency, power, throughput), while the baseline Claude Code only reached 1.07×.
- SOL-Exec: Ranked 1st on 121 of 235 kernels on the public leaderboard, significantly ahead of proprietary and human-tuned entries.
- Chip Design (DPC4): Surpassed the previous SOTA (ArchAgent v2) and the best human design. It delivered a 6.3% performance gain in 16.7 days, compared to ArchAgent v2’s 3.8% gain which took over 40 days.
- SWE-fficiency: Doubled the average speedup achieved by human expert patches.
Limitations and Future Work
- Warmup Overhead: Constructing yardsticks and running initial fidelity tests requires an upfront time investment. Consequently, DuoDiscover is less advantageous for extremely short search budgets where there isn't enough time to amortize the cost of building proxies.
- Yardstick Availability: The system relies on the agent's ability to create a faithful proxy. If a specific optimization target is so deeply coupled with the rest of the system that no fast yardstick can be built, DuoDiscover falls back to standard, slow search.
- Future Directions: The authors suggest extending this "co-discovery" approach to even more diverse domains beyond software and chip design, potentially into experimental sciences where physical lab evaluations are the bottleneck.
Summary updated