Can EDA Tool Feedback Improve Verilog Generation by LLMs?

Jason Blocklove, Shailja Thakur, Benjamin Tan, Hammond Pearce, Siddharth Garg, Ramesh Karri

Abstract

Designing digital hardware using Verilog is a specialized and error-prone task. While Large Language Models (LLMs) have shown promise in code generation, most existing research focuses on "single-shot" generation, which lacks the iterative refinement process used by human engineers. This paper introduces AutoChip, an open-source framework that automates the feedback loop between LLMs and Electronic Design Automation (EDA) tools. By employing a greedy tree search to generate and repair Verilog code based on compiler and simulator feedback, AutoChip enables models to debug their own designs. Our results show that while high-capacity models like GPT-4o thrive with this feedback, a "mixed-model" strategy—combining small, inexpensive models for drafting with high-performance models for final refinement—achieves comparable success rates at a significantly lower cost.

Introduction

In the real world, hardware engineers rarely produce perfect Verilog code on the first attempt. They rely on EDA tools to compile code, run simulations, and identify bugs through error messages. However, most LLM-based hardware generation benchmarks evaluate models on a single-shot basis. This paper addresses this discrepancy by asking: Can we automate the iterative design process so LLMs can debug their own hardware designs without human intervention?

The AutoChip Framework

The researchers developed AutoChip to implement a greedy tree search for Verilog generation and repair. The process follows four main stages:

  1. Generation: Starting from a design prompt, the LLM generates $k$ candidate solutions.
  2. Evaluation: Each candidate is processed through an EDA toolchain (e.g., Icarus Verilog). Candidates are ranked based on their performance: passing all tests is the objective, while failures are ranked by the proportion of correct output samples or the presence of compilation warnings.
  3. Feedback Loop: The code and associated error messages (compilation errors or simulation mismatches) from the best-performing candidate are fed back into the LLM to generate a new set of refined candidates.
  4. Search Depth: This iterative process continues up to a predefined tree depth $d$.
The AutoChip framework uses a tree search to iteratively refine Verilog designs based on tool feedback.

The study also compared prompting styles, finding that "succinct" prompting—providing only the most recent code and errors—achieved similar success to "full context" prompting while significantly reducing token consumption.

Experimental Setup

The framework was evaluated using the VerilogEval benchmark, which includes two distinct subsets:

  • VerilogEval-Machine: Verbose, lower-level problem descriptions generated by GPT-3.5 from existing code.
  • VerilogEval-Human: Abstract, realistic requirements derived from manual reviews of timing diagrams and circuit figures.

Key Results and Insights

The study revealed several critical findings regarding the intersection of model intelligence and tool feedback:

Intelligence and Debugging

Not all models utilize feedback effectively. While GPT-4o showed consistent improvement across iterations, smaller models like GPT-4o-Mini or Claude 3 Haiku often struggled to interpret failure reasons, sometimes failing to improve despite multiple feedback rounds.

The "Mixed-Model" Strategy

A significant breakthrough was the discovery of the "mixed-model" approach. By using a cheap, small model for the initial iterations and switching to a high-performance model like GPT-4o only for the final repair step, the researchers achieved success rates similar to using GPT-4o throughout the entire process.

Cost Efficiency

The economic implications of this approach are substantial. Using feedback with GPT-4o led to a 5.8% increase in successful designs and a 34.2% reduction in total costs compared to zero-shot prompting, as the model found solutions more quickly. The mixed-model approach was even more efficient, resulting in an 89.6% decrease in cost compared to zero-shot GPT-4o without sacrificing performance.

Comparison of token costs for the LLMs evaluated in the study.

Limitations and Future Directions

Despite its success, AutoChip faces several challenges:

  • Error Interpretation: LLMs still struggle to map specific simulation timestamps or complex error messages to underlying logic errors.
  • Design Complexity: The framework currently requires functional testbenches and struggles with the hierarchical module structures common in large-scale hardware design.
  • Search Algorithms: Future iterations could explore more sophisticated search strategies, such as Monte Carlo Tree Search (MCTS), to navigate the design space more effectively.

Conclusion

AutoChip demonstrates that integrating EDA tool feedback into the LLM generation process significantly enhances Verilog design success. By optimizing the choice of models across the design lifecycle, engineers can achieve high-quality hardware generation at a fraction of the traditional computational cost.

Summary updated

BibTeX
@article{blocklove2025can,
  author = {Jason Blocklove and Shailja Thakur and Benjamin Tan and Hammond Pearce and Siddharth Garg and Ramesh Karri},
  title = {{Can EDA Tool Feedback Improve Verilog Generation by LLMs?}},
  journal = {ACM Trans. Design Autom. Electr. Syst.},
  year = {2025},
  doi = {10.1145/3723876},
}
Figure