Agentic Hardware Design as Repository-Level Code Evolution

Cunxi Yu, Chenhui Deng, Nathaniel Pinckney, Brucek KhailanyarXiv 2026 · arXiv

TL;DR

This paper introduces HORIZON, a self-evolving agent framework that treats hardware design not as a one-shot generation task, but as a repository-level code evolution process. By anchoring an LLM agent inside an isolated, version-controlled Git worktree with automated tool feedback, HORIZON achieves a 100% pass rate across multiple challenging RTL design and verification benchmarks.

What Problem Does This Paper Address?

Generating Register-Transfer Level (RTL) hardware descriptions (like Verilog) is notoriously difficult for standard Large Language Models (LLMs). Unlike general software engineering, hardware correctness requires strict cycle-level accuracy, specific reset and interface conventions (e.g., ready-valid protocols), and bit-width precision. A single-turn prompt rarely yields working code because "plausible-looking" Verilog often fails catastrophically during compilation or simulation.

While prior work has successfully used automated feedback loops to evolve software systems (like compilers or SAT solvers), applying this paradigm directly to hardware design artifacts has been a missing link. The authors address this gap by asking: Can hardware design itself be managed as a self-contained, repository-level code evolution process?

Methods

HORIZON shifts the paradigm from prompt engineering to repository engineering. The framework operates as follows:

  1. The Markdown Harness & Project Pack: A user provides a structured Markdown file defining the objective, domain knowledge, and acceptance criteria. A bootstrap agent compiles this into a "project pack"—the control plane that dictates the agent's mission, tools, and evaluation policies.
  2. Git-Native Agent Loop: The agent loop executes entirely within an isolated Git worktree. Git is not just used for bookkeeping; it serves as the agent's core state management and memory substrate. Proposed edits are analyzed via git diff, successful iterations become git commits, and evaluation logs are stored directly in git notes.
  3. Persistent Session Reuse: To keep API costs manageable, HORIZON maintains a persistent model session across iterations. This allows the harness, project pack, and stable source code to be served out of the provider's prompt cache, ensuring that the marginal cost of long-horizon debugging remains low.
Figure 1: Overview of HORIZON. A human-defined Markdown harness is converted into a project pack that specifies the mission, domain skills, evaluator, correctness gate, and git/runtime policy. The resulting task is solved by a self-contained agent loop over git-traced repository states. The executable task instance provides evaluator feedback and reward, while the depth-k trace buffer records version history, state deltas, evaluator outcomes, and runtime summaries.

Datasets

The authors evaluated HORIZON across four major RTL design and verification suites:

  • Verilog-Evalv2: Focuses on HDLBits-style Verilog generation.
  • RTLLM-2.0: Natural language specification-to-RTL tasks.
  • ChipBench: A suite of mixed RTL generation tasks.
  • CVDP (Comprehensive Verilog Design Problems): Evaluated across 9 distinct categories (CIDs 002–016), covering complex tasks like code completion, modification, module reuse, testbench stimulus generation, checker generation, assertion generation, and debugging.

Key Insights and Ideas

  • Git as an Agent Substrate: Version control semantics (branching, diffs, commits) provide a natural, structured environment for agentic exploration. The repository's history effectively becomes a replayable experience trace.
  • The Fallacy of Pass@1: Evaluating hardware agents on their first attempt (Pass@1) severely underestimates their potential. Given an executable harness and the freedom to iterate, agents can systematically debug their way from a near-0% success rate to full convergence.
  • The "Stopping Gate" Effect: When an agent is driven purely by a pass/fail test harness, it stops optimizing the moment the gate opens. For instance, in testbench generation, the agent successfully made the benchmark harness pass, but the resulting design coverage plateaued at 97.9% rather than hitting 100% because the acceptance predicate didn't explicitly demand it.

Key Results

HORIZON achieved a 100% pass rate across all evaluated benchmark suites in a fully hands-free loop.

  • Convergence Trajectories: Simpler suites like RTLLM and Verilog-Eval converged in just 2 iterations. In contrast, the highly challenging CVDP code completion category (CID 002) required a long tail of 82 iterations to resolve stubborn bugs and reach 100% correctness.
Figure 2: Best-so-far pass-rate trajectories over agent iterations. Earlier RTL generation suites saturate within a few iterations, while CVDP exposes longer repair trajectories and clearer differences in convergence difficulty.
  • Token Efficiency: The entire campaign consumed 209.9 million tokens. Crucially, due to HORIZON's session-reuse architecture, approximately 91% of all tokens were cached inputs, drastically reducing the actual API cost of running such an extensive iterative loop.

Limitations and Future Work

Despite the perfect benchmark scores, the authors explicitly state that agentic hardware design is far from solved. They highlight two critical limitations:

  1. Reward Hacking and Over-Solving: Because the agent has full access to simulation logs and error messages, it risks "over-fitting" or customizing the RTL to satisfy the specific, visible test harness rather than implementing robust, generalized design semantics. Future benchmarks need hidden, randomized test suites or formal equivalence checks to audit the solutions.
  2. Feedback Turnaround Latency: While these benchmarks run quickly, real-world Electronic Design Automation (EDA) flows (synthesis, placement, routing, and PPA evaluation) can take hours or days. Running a naive edit-evaluate-repair loop under such delayed and expensive feedback remains a major open research challenge.

Summary updated

Abstract

We present HORIZON, a self-evolving agent framework that treats hardware design as repository-level code evolution. A Markdown harness is compiled into a project pack containing domain knowledge, an executable evaluator, an acceptance predicate, and a git/runtime policy; a hands-free agent loop then evolves an isolated git worktree, using repository operations for state management, tracing, and replay. This extends prior works of repository-scale self-evolution from EDA software systems, to hardware-design artifacts themselves. We evaluate our approach on ChipBench, RTLLM, Verilog-Eval, and nine CVDP categories, achieving 100\% benchmark completion across all suites with a fully hands-free agentic loop. However, we do not claim that agentic AI for hardware design is solved: these benchmarks are controlled proxies for a much broader engineering problem in chip design. Section~\ref{sec:discuss} examines the limitations of the current study and highlights open research challenges.
BibTeX
@misc{yu2026agentic,
  author = {Cunxi Yu and Chenhui Deng and Nathaniel Pinckney and Brucek Khailany},
  title = {{Agentic Hardware Design as Repository-Level Code Evolution}},
  year = {2026},
  eprint = {2606.28279},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.28279},
}
Figure