TL;DR
This paper revisits the VerilogEval benchmark, a crucial tool for evaluating Large Language Models (LLMs) in hardware code generation. By enhancing the benchmark with new tasks (specification-to-RTL), improved failure analysis, and in-context learning support, the authors demonstrate significant progress in LLM capabilities for hardware design, with open-source models now rivaling closed-source counterparts.
What Problem Does This Paper Address?
The field of using Large Language Models (LLMs) for hardware code generation, specifically in Verilog, is rapidly evolving. However, evaluating these models has been challenging due to a lack of comprehensive benchmarks and a focus on natural language and software code in LLM training. The original VerilogEval benchmark, released in late 2023, provided a starting point, but the rapid pace of LLM development necessitated an update. This paper addresses the need for a more robust and up-to-date evaluation framework that can capture the nuances of modern instruction-tuned LLMs and their performance on more realistic hardware design tasks, moving beyond simple code completion.
Methods
The authors introduce VerilogEval v2, an improved version of the original benchmark. Key enhancements include:
- Specification-to-RTL Task Support: Moving beyond code completion, VerilogEval v2 now supports translating high-level specifications into Verilog code, better aligning with how instruction-tuned LLMs are designed.
- In-Context Learning (ICL) Support: The benchmark now allows for the inclusion of ICL examples within prompts, enabling the evaluation of how LLMs learn from few-shot examples.
- Failure Classification: A detailed classification system for various types of Verilog compilation and runtime errors has been implemented. This provides crucial insights into why models fail, guiding future prompt engineering and model development.
- Makefile-based Evaluation: The infrastructure has been refactored to use Makefiles, allowing for easier scaling, more flexible evaluation sweeps, and improved human inspection of results.
Datasets
The primary dataset used is the VerilogEval dataset, which was originally sourced from HDLBits. This dataset was adapted and expanded for VerilogEval v2 to include:
- VerilogEval-human: Human-created problem descriptions, considered more aligned with real-world code generation scenarios.
- Specification-to-RTL Task Data: The existing 156 problems from VerilogEval were converted into a specification-to-RTL format.
- In-Context Learning Examples: Various examples were curated for both code completion and specification-to-RTL tasks, with varying "shot" counts (1-shot, 2-shot, 3-shot).
Key Insights and Ideas
- Open Models are Catching Up: The performance gap between closed-source (e.g., GPT-40) and open-source models (e.g., Llama3.1 405B) has significantly narrowed, with Llama3.1 405B achieving state-of-the-art results on some tasks.
- Prompt Tuning is Crucial (and Complex): In-context learning can dramatically impact LLM performance, sometimes leading to significant improvements and other times causing regressions (e.g., Llama3 70B). This highlights the need for careful prompt engineering tailored to specific models and tasks.
- Specification-to-RTL is More Promising: LLMs generally perform better when tasked with generating Verilog from specifications rather than just completing code snippets, likely due to the increased flexibility in defining interfaces.
- Failure Analysis is Key: Understanding the types of errors LLMs make (e.g., "Reg Declared as Wire," "Unable to Bind Wire/Reg 'clk'") is essential for targeted improvements and developing better LLM-based hardware design tools.
- Domain-Specific Models Offer Efficiency: Smaller, specialized models like RTL-Coder 6.7B can achieve competitive performance with significantly fewer parameters, demonstrating the value of domain-specific fine-tuning.
Key Results
- GPT-40 achieved a 63% pass rate on specification-to-RTL tasks.
- Llama3.1 405B achieved a 58% pass rate on specification-to-RTL tasks, nearly matching GPT-40.
- RTL-Coder 6.7B achieved an impressive 34% pass rate on specification-to-RTL tasks, demonstrating strong performance for its size.
- In code completion, Llama3.1 405B achieved a 57.0% pass rate (0-shot), while GPT-40 reached 61% with 1-shot ICL.
- The paper observed significant improvements in LLM performance over the past year, with newer models consistently outperforming older ones.
- Failure analysis revealed common issues like "Reg Declared as Wire" and "Unable to Bind Wire/Reg 'clk'," which can be mitigated through prompt tuning.
Limitations and Future Work
- Dataset Complexity: The authors note that the VerilogEval dataset's complexity is becoming limited for the most advanced LLMs and agent-based approaches.
- Agent-based Approaches: Future work should explore agent-based LLM systems that can handle multi-turn interactions, planning, and debugging, mimicking human design workflows.
- Broader Hardware Design Tasks: The research community needs benchmarks that extend beyond RTL generation to cover other critical hardware design tasks like testbench creation, assertion generation, and documentation.
- Model-Specific Tuning: The impact of prompt tuning is highly model-dependent, necessitating further research into tailored strategies for different LLMs.
- Scalability of Agentic Systems: While agentic approaches show promise, their computational demands and the need for specialized LLMs for different subtasks require further investigation.
Summary updated