Revisiting VerilogEval: Newer LLMs, In-Context Learning, and Specification-to-RTL Tasks

Nathaniel Ross Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren, Brucek Khailany

TL;DR

This paper revisits the VerilogEval benchmark for hardware code generation, introducing significant improvements to its infrastructure and evaluation capabilities. By incorporating specification-to-RTL tasks, in-context learning, and automated failure analysis, the study evaluates recent large language models (LLMs), revealing that open-source models like Llama3.1 are now competitive with commercial giants, and that prompt engineering remains crucial for optimal performance.

What Problem Does This Paper Address?

The field of using Large Language Models (LLMs) for hardware code generation, specifically in Verilog, is rapidly evolving. However, existing benchmarks like the original VerilogEval were limited: they primarily focused on code completion, lacked detailed failure analysis, and didn't fully support newer prompting techniques like in-context learning (ICL). Furthermore, the pace of LLM development means that benchmarks quickly become outdated. This paper addresses these limitations by enhancing VerilogEval to be a more comprehensive and up-to-date evaluation framework, enabling a deeper understanding of LLM capabilities and the impact of various prompting strategies in hardware design.

Methods

The authors introduce VerilogEval v2, an improved version of the original benchmark. Key enhancements include:

  • Specification-to-RTL (Spec-to-RTL) Task Support: Moving beyond just code completion, VerilogEval v2 now supports generating Verilog code directly from natural language specifications, better aligning with how many modern LLMs are trained (instruction-tuned).
  • In-Context Learning (ICL) Examples: The benchmark now allows for the inclusion of ICL examples within prompts. This technique, where a few examples of input-output pairs are provided to the LLM, has been shown to significantly improve performance on various tasks. The number of ICL examples (shots) is also parameterized for flexible evaluation.
  • Automated Failure Classification: Instead of just a pass/fail result, VerilogEval v2 automatically categorizes failures into specific types (e.g., compile-time errors, runtime errors, syntax issues, incorrect logic). This provides crucial insights into why models fail and guides prompt engineering efforts.
  • Makefile-based Evaluation: The infrastructure has been refactored to use Makefiles, making it more scalable, easier to manage, and facilitating human inspection of results.

Datasets

The primary dataset used is the VerilogEval dataset, which was originally composed of 156 problems. This dataset was adapted and extended for VerilogEval v2. The paper mentions two versions of problem descriptions:

  • VerilogEval-machine: Descriptions generated by GPT-3.5.
  • VerilogEval-human: Human-created descriptions from HDLBits, considered more aligned with real-world code generation scenarios.

The paper also notes that 14 problems in the dataset had their descriptions or test benches revised for improved clarity and consistency.

Key Insights and Ideas

  • Open-Source Models Catching Up: The performance gap between leading commercial LLMs and open-source models has significantly narrowed. Models like Llama3.1 405B are now achieving pass rates comparable to top commercial models like GPT-40.
  • Prompt Engineering is Paramount: The effectiveness of LLMs for hardware code generation is highly sensitive to prompt design. Subtle changes in prompts, including the use of ICL examples, can lead to substantial improvements or degradations in performance, varying significantly across different models.
  • Spec-to-RTL Offers More Flexibility: Generating code from specifications (Spec-to-RTL) generally yields better results than pure code completion. This is likely because it allows LLMs more freedom to define interfaces and structures, which can be more forgiving than strict code completion tasks.
  • Domain-Specific Models Shine: Smaller, specialized models like RTL-Coder 6.7B demonstrate impressive performance relative to their size, highlighting the potential of fine-tuning for specific tasks.
  • ICL's Nuanced Impact: While ICL often improves performance, its effect is not uniform. Some models (like Llama3 70B) can even degrade in performance with ICL, suggesting that the quality and relevance of ICL examples are critical.

Key Results

  • State-of-the-Art Performance: GPT-40 achieved a 63% pass rate on specification-to-RTL tasks. Llama3.1 405B achieved a 58% pass rate on code completion, nearly matching GPT-40.
  • Domain-Specific Efficiency: RTL-Coder 6.7B achieved a 34% pass rate on code completion, demonstrating strong performance for its size.
  • Model Evolution: GPT-40 and GPT-4 Turbo showed significant improvements over GPT-4. Llama3.1 405B and 70B also showed notable gains compared to their predecessors.
  • ICL Impact Varies: For code completion, GPT-40 saw a ~5% improvement with 1-shot ICL, while Llama3.1 405B showed minimal improvement. For Spec-to-RTL, Llama3.1 70B showed significant improvement with ICL.
  • Failure Analysis Insights: The automated classification revealed common failure modes like "Reg Declared as Wire" (confusing registers and wires) and missing "endmodule" keywords, which can be addressed through prompt tuning.

Limitations and Future Work

  • Dataset Complexity: The authors acknowledge that the current VerilogEval dataset's complexity may be becoming limited for the most advanced LLMs and agentic approaches.
  • Agentic Approaches: Future work should focus on developing more sophisticated agent-based LLM systems that can handle multi-turn interactions, planning, and debugging, mimicking human design workflows.
  • Broader Hardware Design Tasks: The research community needs benchmarks that extend beyond RTL generation to cover other critical hardware design tasks like testbench creation, assertion generation, and documentation.
  • Benchmark Evolution: Future benchmarks should present more realistic and complex design problems that require reasoning and flexibility, rather than overly descriptive prompts, to truly challenge advanced AI agents.

Summary updated

Abstract

The application of large-language models (LLMs) to digital hardware code generation is an emerging field. Most LLMs are primarily trained on natural language and software code. Hardware code, such as Verilog, represents only a small portion of the training data and few hardware benchmarks exist. To address this gap, the open-source VerilogEval benchmark was released in 2023, providing a consistent evaluation framework for LLMs on code completion tasks. It was tested on state-of-the-art models at the time including GPT-4. However, VerilogEval and other Verilog generation benchmarks lack failure analysis and, in present form, are not conducive to exploring prompting techniques. Also, since VerilogEval's release, both commercial and open-source models have seen continued development. In this work, we evaluate new commercial and open-source models of varying sizes against an improved VerilogEval benchmark suite. We enhance VerilogEval's infrastructure and dataset by automatically classifying failures, introduce new prompts for supporting in-context learning (ICL) examples, and extend the supported tasks to specification-to-RTL translation. We find a measurable improvement in commercial state-of-the-art models, with GPT-4 Turbo achieving a 59% pass rate on spec-to-RTL tasks. We also study the performance of open-source and domain-specific models that have emerged, and demonstrate that models can benefit substantially from ICL. We find that recently-released Llama 3.1 405B achieves a pass rate of 58%, effectively matching that of GPT-4 Turbo, and that the much smaller domain-specific RTL-Coder 6.7B models achieve an impressive 37% pass rate. However, prompt engineering is key to achieving good pass rates, and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is key to continued model development and deployment.
BibTeX
@article{pinckney2025revisiting,
  author = {Nathaniel Ross Pinckney and Christopher Batten and Mingjie Liu and Haoxing Ren and Brucek Khailany},
  title = {{Revisiting VerilogEval: Newer LLMs, In-Context Learning, and Specification-to-RTL Tasks}},
  journal = {ACM Trans. Design Autom. Electr. Syst.},
  year = {2025},
  url = {https://openreview.net/forum?id=cUKq79CLkO},
}
Figure