Language models that write hardware code, such as Verilog circuit descriptions, now improve mainly by looping: write a design, simulate it, read the failure, patch it. Two findings from separate literatures do not fit together. Small 7B models trained specifically on Verilog with execution-based rewards match or beat 671B general models when producing a design in one pass. Yet the one system that turns the write–simulate–patch loop into a proper search tree, CodeTree, reports sharply limited gains when its agents run on an 8B model instead of GPT-4o. We propose that the small model is not failing at writing or patching. It fails at judging, meaning deciding which partial repair is worth continuing and which to abandon. If that judgment is taken from the simulator rather than from the model, a compact specialized model should benefit from tree search as much as a frontier model does.
Concretely, we would build a search over repair states for Verilog generation. A compact domain model proposes alternative designs and patches. Branching, refining, and abandoning are decided by a cheap progress score computed from waveform comparisons against a testbench the system writes for itself. The score needs no language model. The central experiment swaps which component is large or small: the generator, the judge, or both. This isolates why small-model tree search has failed, and it tests whether the fix also beats simply sampling many designs from the same model at equal cost.
Core Claim
Tree search over code repairs needs two capabilities. One is generating good alternatives. The other is scoring intermediate states well enough to spend the budget on the right branches. CodeTree assigns both to LLM agents. A Thinker proposes strategies, a Solver implements them, a Debugger patches them, and a Critic chooses among Refine, Abort, and Accept. It reaches 43.0% on CodeContests with GPT-4o, against 32.7% for resampling and 41.8% for linear refinement, but gains little with Llama-3.1-8B. Specialized compact models such as QiMeng-CodeV-R1 show the generation capability can be bought cheaply in a narrow domain. It reports 72.9% pass@1 on RTLLM v1.1 and 68.6% on VerilogEval v2 at 7B. Work on internalized backtracking, such as Self-Verifying Reflection Helps Transformers with CoT Reasoning, shows that even tiny transformers can run depth-first backtracking once verification is reliable.
Our claim is that the Critic is the weak link for small models. Hardware is a domain where the Critic can be replaced by a signal that is almost free and far more discriminative than the model's opinion: how much of the simulated output waveform matches the expected behavior, and at which clock cycle it first diverges. What would be new is showing that execution-grounded judgment removes the scale dependence CodeTree observed. The resulting 7B search system should close most of the gap to frontier multi-agent systems at a fraction of their token cost, and beat best-of-N sampling from the same model at matched simulator budget.
Research Program
The system has three roles. We keep them separate because they need different inputs, not because more agents are better.
The first role is the generator, a specialized compact model; CodeV-R1-7B is the primary backbone. It is prompted in two modes. In strategy mode it proposes a few structurally different implementations, for example a state machine versus a counter-based datapath. In patch mode it receives a failing design plus a short textual waveform window around the first mismatching cycle. That window format is borrowed from MAGE, which extracts the cycle of divergence for its debug agent.
The second role is the testbench writer. It produces stimulus and expected outputs from the natural-language specification alone, as MAGE's testbench agent does. This is essential for honesty. The benchmark's golden testbench is never seen during search. Only the final single submission is graded against it, so pass@1 remains a real single-attempt number.
The third role is the controller. It maintains the tree and needs no language model. For a candidate design $c$ simulated against the self-written testbench, let $m(c)$ be the fraction of checked (cycle, output-signal) pairs that match, and let $\tau(c)$ be the first mismatching cycle divided by the simulation length. The progress score is
$$ s(c) = \mathbb{1}[\text{compiles}(c)] \cdot \big(m(c) + \lambda\, \tau(c)\big), $$
where $\lambda$ is a small weight that rewards pushing divergence later in time. The controller expands the open node with the highest score, and aborts a branch after a fixed number of patches that fail to raise $s$. It also aborts immediately when a child breaks checks its parent passed. That second rule targets the "new error trap" documented in MEIC, where fixing functional logic reintroduces syntax errors and the edits cycle. The controller accepts once all self-written checks pass. Any remaining budget goes to a second, independently sampled testbench to guard against a wrong first testbench.
The design contrasts with each neighbor. MEIC and ReChisel are linear loops with a one-step rollback or a rule-triggered escape. ReChisel lifts Claude 3.5 Sonnet from 33.33% to 84.98% pass@1 on its Chisel benchmark, but it never holds competing hypotheses in parallel. MAGE samples diversely at the root and then debugs each candidate linearly with frontier models. VerilogCoder plans over a task graph and traces waveforms through the syntax tree, but executes linearly and uses about 13 times the tokens of plain GPT-4 Turbo. a1 shows that execution-triggered backtracking matters: removing it costs 21.1 points on AIME24. However, a1 builds that backtracking into one monolithic 32B model for math. PGTS learns a branch/backtrack controller but scores nodes with token likelihood, and it struggles on hard science. That result warns against model-internal scores and motivates grounding the score in execution. A PGTS-style learned controller that takes our waveform features as input is a natural second stage. We would pursue it only if the hand-designed score leaves measurable headroom.
A secondary transfer study repeats the design for GPU kernels in Triton, with AutoTriton-8B as generator. There the progress score is the fraction of output elements numerically matching the PyTorch reference that defines the task. The purpose is to test whether the mechanism depends on Verilog specifically or on having a graded execution signal.
Baselines
The naive reference point is the specialized generator alone, run once with the authors' decoding settings. For CodeV-R1-7B, the paper reports 72.9% pass@1 on RTLLM v1.1 and 68.6% on VerilogEval v2. We re-run it in our harness so that any difference reflects the harness rather than the benchmark version. We also report the base Qwen2.5-Coder-7B under the full search system, which tells us how much of any gain requires specialization.
The strongest compute-matched competitor is best-of-N sampling from the same 7B model. The N samples are filtered by the same self-written testbench, and the highest-scoring candidate is submitted. It uses identical generator, testbench writer, and simulator, and N is set so total generated tokens and simulator calls equal the tree's budget. Its hyperparameters are tuned on the same development split with the same number of trials. The tree must beat it by a margin exceeding run-to-run variance at two or more budgets to count as a win.
The linear-refinement competitor is a MEIC-style loop run on the same 7B model and the same waveform feedback, with rollback to the previous best version. It is matched on budget. The frontier ceiling is MAGE with Claude 3.5 Sonnet, which reports 95.7% on VerilogEval-Human v2, and VerilogCoder at 94.2% on the same benchmark. The file does not state whether "VerilogEval v2" in CodeV-R1 and "VerilogEval-Human v2" in MAGE are exactly the same split. We will evaluate all systems on one fixed split rather than mix reported numbers. We do not expect to beat these frontier systems on raw accuracy. Success against them means approaching within a few points at a small fraction of their token cost, which we measure directly.
The key ablation is a CodeTree-style LLM Critic run on the same 7B model and the same tree. It isolates the one ingredient we claim matters, the source of judgment. It sits inside the factorial experiment below.
Why not just…?
Why not just sample many designs from the specialized model and keep the best? This is the most serious threat. MAGE's own large gain comes partly from high-temperature root sampling, and at small budgets best-of-N may simply win. Evidence cuts both ways. VerilogEval notes that fine-tuning raises pass@1 but can lower pass@5 and pass@10, which means specialized models produce less diverse samples, the regime where resampling saturates. CodeTree also found tree search beat resampling by ten points with GPT-4o. The budget-matched comparison settles the question. If best-of-N matches the tree at every budget, we will report that the search adds nothing beyond a good filter and narrow the claim to the filter itself.
Why not just refine linearly with rollback, as MEIC does? Linear loops plateau within three to five iterations according to Yan et al., and MEIC documents cycling on dense arithmetic modules. Our prediction is specific. The tree's advantage over the linear baseline should concentrate on problems where the linear loop's score stops rising or oscillates. If gains are instead spread uniformly, the mechanism is not the one we claim.
Why not just prompt a frontier model inside MAGE or VerilogCoder? On VerilogEval-Human v2 those systems already sit near 95%, so accuracy there is nearly saturated. The case for a compact system is cost and local deployment. VerilogCoder's roughly 13-fold token overhead and VeriGen's measured latency of about 2 seconds versus 10 seconds for GPT-4 show what is at stake. On harder designs, frontier general models do not dominate: CHORUS reports DeepSeek-R1 at 74.5% against a 4B specialist at 88.0% on CVDP-ECov. We therefore emphasize RTLLM, where headroom remains, and report accuracy per million tokens.
Why not just keep CodeTree's LLM Critic but specialize it too? This is a fair variant. We include a Critic fine-tuned on simulator-labeled repair states as a fourth cell of the factorial. If it matches the simulator score, the contribution becomes "the Critic needs domain grounding," which is still useful but less interesting than the claim that judgment can be taken out of the model entirely.
Proposed Experiments
The primary benchmarks are RTLLM v1.1 (30 designs up to a simplified RISC-V CPU), VerilogEval-Human v2 (156 tasks), and MEIC's set of 178 injected RTL errors, reported as syntax and functional fix rates. For MEIC's set, the reported references are 78% functional fixes for MEIC with GPT-4 and 66% for retrieval or fine-tuning. The Triton transfer uses TRITONBENCH-T, where AutoTriton-8B reports 39.16% execution accuracy in single-pass generation. Every system is scored by pass@1 against the golden testbench on one submitted design. We also report generated tokens, simulator calls, and wall-clock time.
The first decisive check is the factorial experiment. It crosses generator size (CodeV-R1-7B versus a frontier model) with judge type (7B LLM Critic, frontier LLM Critic, simulator progress score) inside the identical tree. If our diagnosis is right, the 7B-generator/7B-Critic cell reproduces CodeTree's weak small-model gains. Switching only the judge to the simulator score should recover most of the gap to the all-frontier cell. Switching only the generator to frontier scale should recover much less.
The second check is predictive. On logged search trees, we measure how well each judge's scores at intermediate nodes predict whether that branch eventually yields a correct design. Discrimination is summarized as area under the ROC curve. The simulator score should clearly out-predict the 7B Critic; if it does not, the mechanism is wrong even if accuracy improves.
The third check is the budget-matched comparison against best-of-N and linear refinement at three budgets. We stratify problems by whether the linear loop plateaued, and look for gains concentrated on the plateaued ones.
The fourth check measures how often the self-written testbench is wrong, meaning it disagrees with the golden testbench on the reference design. We then quantify how much accuracy is lost to testbench errors. This bounds the method's ceiling and tells us whether the next investment belongs in the testbench writer rather than the search.
References
- CodeTree: Agent-guided Tree Search for Code Generation with Large Language Models
- QiMeng-CodeV-R1: Reasoning-Enhanced Verilog Generation
- Self-Verifying Reflection Helps Transformers with CoT Reasoning
- MAGE: A Multi-Agent Engine for Automated RTL Code Generation
- MEIC: Re-thinking RTL Debug Automation using LLMs
- ReChisel: Effective Automatic Chisel Code Generation by LLM with Reflection
- VerilogCoder: Autonomous Verilog Coding Agents with Graph-based Planning and Abstract Syntax Tree (AST)-based Waveform Tracing Tool
- a1: Steep Test-time Scaling Law via Environment Augmented Generation
- Policy Guided Tree Search for Enhanced LLM Reasoning
- Invited Paper: VerilogEval: Evaluating Large Language Models for Verilog Code Generation
- CodeTransOcean: A Comprehensive Multilingual Benchmark for Code Translation
- VeriGen: A Large Language Model for Verilog Code Generation
- CHORUS: Complementary Experts for High-Coverage Testbench Stimulus Generation