Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation

Nathaniel Ross Pinckney, Christopher Batten, Mingjie Liu, Haoxing Ren, Brucek Khailany

TL;DR

This paper revisits the VerilogEval benchmark, a crucial tool for evaluating Large Language Models (LLMs) in hardware code generation. By enhancing the benchmark with new tasks (specification-to-RTL), improved failure analysis, and in-context learning support, the authors demonstrate significant progress in LLM capabilities for hardware design, with open-source models now rivaling closed-source counterparts.

What Problem Does This Paper Address?

The field of using Large Language Models (LLMs) for hardware code generation, specifically in Verilog, is rapidly evolving. However, evaluating these models has been challenging due to a lack of comprehensive benchmarks and a focus on natural language and software code in LLM training. The original VerilogEval benchmark, released in late 2023, provided a starting point, but the rapid pace of LLM development necessitated an update. This paper addresses the need for a more robust and up-to-date evaluation framework that can capture the nuances of modern instruction-tuned LLMs and their performance on more realistic hardware design tasks, moving beyond simple code completion.

Methods

The authors introduce VerilogEval v2, an improved version of the original benchmark. Key enhancements include:

  • Specification-to-RTL Task Support: Moving beyond code completion, VerilogEval v2 now supports translating high-level specifications into Verilog code, better aligning with how instruction-tuned LLMs are designed.
  • In-Context Learning (ICL) Support: The benchmark now allows for the inclusion of ICL examples within prompts, enabling the evaluation of how LLMs learn from few-shot examples.
  • Failure Classification: A detailed classification system for various types of Verilog compilation and runtime errors has been implemented. This provides crucial insights into why models fail, guiding future prompt engineering and model development.
  • Makefile-based Evaluation: The infrastructure has been refactored to use Makefiles, allowing for easier scaling, more flexible evaluation sweeps, and improved human inspection of results.

Datasets

The primary dataset used is the VerilogEval dataset, which was originally sourced from HDLBits. This dataset was adapted and expanded for VerilogEval v2 to include:

  • VerilogEval-human: Human-created problem descriptions, considered more aligned with real-world code generation scenarios.
  • Specification-to-RTL Task Data: The existing 156 problems from VerilogEval were converted into a specification-to-RTL format.
  • In-Context Learning Examples: Various examples were curated for both code completion and specification-to-RTL tasks, with varying "shot" counts (1-shot, 2-shot, 3-shot).

Key Insights and Ideas

  • Open Models are Catching Up: The performance gap between closed-source (e.g., GPT-40) and open-source models (e.g., Llama3.1 405B) has significantly narrowed, with Llama3.1 405B achieving state-of-the-art results on some tasks.
  • Prompt Tuning is Crucial (and Complex): In-context learning can dramatically impact LLM performance, sometimes leading to significant improvements and other times causing regressions (e.g., Llama3 70B). This highlights the need for careful prompt engineering tailored to specific models and tasks.
  • Specification-to-RTL is More Promising: LLMs generally perform better when tasked with generating Verilog from specifications rather than just completing code snippets, likely due to the increased flexibility in defining interfaces.
  • Failure Analysis is Key: Understanding the types of errors LLMs make (e.g., "Reg Declared as Wire," "Unable to Bind Wire/Reg 'clk'") is essential for targeted improvements and developing better LLM-based hardware design tools.
  • Domain-Specific Models Offer Efficiency: Smaller, specialized models like RTL-Coder 6.7B can achieve competitive performance with significantly fewer parameters, demonstrating the value of domain-specific fine-tuning.

Key Results

  • GPT-40 achieved a 63% pass rate on specification-to-RTL tasks.
  • Llama3.1 405B achieved a 58% pass rate on specification-to-RTL tasks, nearly matching GPT-40.
  • RTL-Coder 6.7B achieved an impressive 34% pass rate on specification-to-RTL tasks, demonstrating strong performance for its size.
  • In code completion, Llama3.1 405B achieved a 57.0% pass rate (0-shot), while GPT-40 reached 61% with 1-shot ICL.
  • The paper observed significant improvements in LLM performance over the past year, with newer models consistently outperforming older ones.
  • Failure analysis revealed common issues like "Reg Declared as Wire" and "Unable to Bind Wire/Reg 'clk'," which can be mitigated through prompt tuning.

Limitations and Future Work

  • Dataset Complexity: The authors note that the VerilogEval dataset's complexity is becoming limited for the most advanced LLMs and agent-based approaches.
  • Agent-based Approaches: Future work should explore agent-based LLM systems that can handle multi-turn interactions, planning, and debugging, mimicking human design workflows.
  • Broader Hardware Design Tasks: The research community needs benchmarks that extend beyond RTL generation to cover other critical hardware design tasks like testbench creation, assertion generation, and documentation.
  • Model-Specific Tuning: The impact of prompt tuning is highly model-dependent, necessitating further research into tailored strategies for different LLMs.
  • Scalability of Agentic Systems: While agentic approaches show promise, their computational demands and the need for specialized LLMs for different subtasks require further investigation.

Summary updated

Abstract

The application of large-language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development. In this work, we evaluate new commercial and open models since VerilogEval's original release-including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder-against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate. Additionally, we enhance VerilogEval's infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.
BibTeX
@article{pinckney2025revisiting,
  author = {Nathaniel Ross Pinckney and Christopher Batten and Mingjie Liu and Haoxing Ren and Brucek Khailany},
  title = {{Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation}},
  journal = {ACM Trans. Design Autom. Electr. Syst.},
  year = {2025},
  url = {https://openreview.net/forum?id=cUKq79CLkO},
}
Figure