Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback

Jason Blocklove, Shailja Thakur, Benjamin Tan, Hammond Pearce, Siddharth Garg, Ramesh Karri

TL;DR

This paper introduces AutoChip, a framework that uses Electronic Design Automation (EDA) tool feedback to iteratively improve Verilog code generated by Large Language Models (LLMs). They demonstrate that this feedback-driven approach significantly enhances the success rate of Verilog generation, especially for more complex designs, and can even reduce the overall cost compared to zero-shot LLM generation.

What Problem Does This Paper Address?

Designing digital hardware using Hardware Description Languages (HDLs) like Verilog is a complex and time-consuming process, prone to errors. While LLMs show promise in automatically generating Verilog code, existing methods often rely on a "zero-shot" approach, where the LLM generates code in a single attempt without iterative refinement. This doesn't mirror real-world hardware design, which involves engineers using tools to debug and improve their code. The authors aim to bridge this gap by automating the feedback loop, allowing LLMs to learn from EDA tool outputs (compilation messages, simulation results) to fix their own generated Verilog code.

Methods

The core of the paper is the AutoChip framework. It takes a natural language description of a circuit and an accompanying testbench as input. AutoChip then:

  1. Generates multiple candidate Verilog modules from an LLM for a given prompt.
  2. Evaluates these candidates using Verilog compilers and simulators.
  3. Ranks candidates based on compilation success and testbench simulation results.
  4. Feeds the feedback (compilation errors, simulation mismatches) from the best-performing candidate back to the LLM.
  5. Iteratively refines the Verilog code by prompting the LLM to fix identified issues.

This process is structured as a greedy tree search, where the best candidate at each step is used to generate the next set of candidates, up to a defined maximum depth. Two feedback strategies were explored: "full-context" (all conversation history) and "succinct" (only the most recent feedback), with "succinct" proving more efficient due to LLM context window limitations.

Datasets

The primary dataset used for evaluation is VerilogEval v1. This dataset comprises prompts and testbenches derived from HDLBits, a platform for Verilog practice. VerilogEval includes both "Machine" (LLM-generated prompts) and "Human" (manually curated prompts) benchmark sets, allowing for evaluation of LLM performance on different prompt qualities.

Key Insights and Ideas

  • EDA tool feedback is crucial: Unlike zero-shot generation, feedback from compilers and simulators significantly improves the success rate of LLM-generated Verilog, especially for more complex designs.
  • LLM capability matters: The effectiveness of feedback is highly dependent on the LLM's inherent capabilities. More advanced models like GPT-40 benefit more consistently from this feedback loop.
  • Cost-effectiveness of mixed models: Combining less expensive LLMs for initial iterations with a powerful model like GPT-40 for a final refinement pass can achieve comparable success rates to using GPT-40 alone, but at a significantly reduced cost.
  • Tree search parameters influence results: Increasing the number of candidates (k) and the search depth (d) generally improves success rates, with k appearing to have a larger impact.
  • Problem type influences LLM performance: LLMs excel at basic Verilog syntax and logic but struggle more with abstract problems like Karnaugh maps, state machine design, and interpreting waveform data.

Key Results

  • Improved Success Rates: EDA tool feedback led to a 5.8% increase in successful designs and a 34.2% decrease in cost compared to zero-shot prompting with GPT-40.
  • Cost Reduction with Mixed Models: Using a mixed-model approach (smaller LLMs followed by GPT-40) achieved similar success to GPT-40 alone but incurred 41.9% lower cost (and an overall 89.6% cost reduction compared to zero-shot GPT-40).
  • GPT-40's Advantage: GPT-40 consistently showed improved results with feedback, while smaller models like GPT-3.5-Turbo and Claude 3 Haiku showed inconsistent benefits from feedback alone.
  • Parameter Impact: Increasing candidates (k) and depth (d) generally led to higher success rates, though the gains from increasing depth tended to plateau.

Limitations and Future Work

  • LLM Dependency: The effectiveness of feedback is heavily model-dependent. Smaller models struggle to correlate error messages with design aspects without explicit human guidance. Future work could involve pre-instructing LLMs on specific error messages.
  • Parameter Optimization: While k and d were explored, finding the ideal combination for different problem types remains an open question. Future work could explore more advanced search methodologies like Monte Carlo Tree Search (MCTS).
  • Abstract Problems: LLMs still struggle with highly abstract problems (e.g., Karnaugh maps, state machines, waveform interpretation), especially in the "Human" benchmark set.
  • Cost-Complexity Relationship: The exact relationship between model complexity, cost, and performance is not fully quantified.

The authors provide AutoChip as an open-source, extendable framework to facilitate further research in this area, enabling evaluation of new LLMs and benchmarks.

Summary updated

Abstract

Traditionally, digital hardware designs are written in the Verilog hardware description language (HDL) and debugged manually by engineers. This can be time-consuming and error-prone for complex designs. Large Language Models (LLMs) are emerging as a potential tool to help generate fully functioning HDL code, but most works have focused on generation in the single-shot capacity: i.e., run and evaluate, a process that does not leverage debugging and, as such, does not adequately reflect a realistic development process. In this work, we evaluate the ability of LLMs to leverage feedback from electronic design automation (EDA) tools to fix mistakes in their own generated Verilog. To accomplish this, we present an open-source, highly customizable framework, AutoChip, which combines conversational LLMs with the output from Verilog compilers and simulations to iteratively generate and repair Verilog. To determine the success of these LLMs we leverage the VerilogEval benchmark set. We evaluate four state-of-the-art conversational LLMs, focusing on readily accessible commercial models. EDA tool feedback proved to be consistently more effective than zero-shot prompting only with GPT-4o, the most computationally complex model we evaluated. In the best case, we observed a 5.8% increase in the number of successful designs with a 34.2% decrease in cost over the best zero-shot results. Mixing smaller models with this larger model at the end of the feedback iterations resulted in equally as much success as with GPT-4o using feedback, but incurred 41.9% lower cost (corresponding to an overall decrease in cost over zero-shot by 89.6%).
BibTeX
@article{blocklove2025automatically,
  author = {Jason Blocklove and Shailja Thakur and Benjamin Tan and Hammond Pearce and Siddharth Garg and Ramesh Karri},
  title = {{Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback}},
  journal = {ACM Trans. Design Autom. Electr. Syst.},
  year = {2025},
  doi = {10.1145/3723876},
}
Figure