Abstract
Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification—specifically the detection and diagnosis of corner-case design bugs—is critically dependent on the quality of these assertions. With the rise of generative AI, there is renewed interest in using Large Language Models (LLMs) to generate functional and security assertions from design source code. This paper introduces AssertionBench, a comprehensive set of designs and assertions used to quantify the effectiveness of commercial-off-the-shelf (COTS) LLMs. Our findings suggest that COTS LLMs are not yet ready for practical adoption due to high rates of syntactically and semantically incorrect outputs. To address this, we propose AssertionLLM, a first-of-its-kind model fine-tuned specifically for assertion generation, which demonstrates significant improvements in accuracy over general-purpose models.
Introduction
In modern System-on-Chip (SoC) design, assertions serve as the mathematical rules that ensure a design behaves correctly and securely. For instance, a simple assertion might dictate that a 'grant' signal must follow a 'request' signal within a specific timeframe. Despite their importance, writing high-quality assertions remains a manual bottleneck requiring deep domain expertise. While automated tools exist, they often struggle to scale or produce redundant properties.
As Large Language Models (LLMs) transform software engineering, the hardware community is exploring whether these models can automate assertion generation. This study evaluates the current state of the art and proposes a specialized path forward through domain-specific fine-tuning.
AssertionBench: A New Standard for Evaluation
To rigorously test LLM capabilities, we developed AssertionBench, a suite of 100 Verilog designs sourced from OpenCores. The benchmark is characterized by:
- Diversity: It includes combinational and sequential designs, ranging from simple arbiters to complex components like Floating Point Units (FPUs) and Ethernet controllers.
- Scale: Design sizes vary from 10 to 1,150 lines of code.
- Ground Truth: Each design is paired with an average of 4.8 formally verified assertions to serve as the gold standard for evaluation.
Methodology
Our research followed a two-pronged approach to evaluate and improve assertion generation:
1. Evaluation of COTS Models
We tested several general-purpose models, including GPT-3.5, GPT-4o, CodeLLaMa 2, and LLaMa 3-70B. We utilized In-Context Learning (ICL), providing the models with 1-shot or 5-shot examples of Verilog code and corresponding SystemVerilog Assertions (SVA) to guide their output.
2. Development of AssertionLLM
Recognizing the limitations of general-purpose models, we created AssertionLLM. We fine-tuned CodeLLaMa 2 and LLaMa 3-70B on a specialized dataset of Verilog designs and verified assertions to better capture hardware-specific logic.
3. Formal Verification Loop
To move beyond simple text-matching metrics, we employed the Cadence JasperGold formal verification engine. This allowed us to determine if AI-generated assertions were mathematically valid or if they triggered a Counter-Example (CEX), indicating a logical flaw.
Key Insights and Results
The Limitations of COTS Models
Off-the-shelf models are currently unsuitable for production-level hardware verification. On average, COTS models achieved no more than 44% accuracy. A staggering 63% of their assertions were semantically incorrect, and 33% were syntactically broken. Interestingly, GPT-4o emerged as the most consistent COTS model, producing the fewest syntax errors.
The Impact of Fine-Tuning
AssertionLLM demonstrated that domain-specific training is essential. Fine-tuning CodeLLaMa 2 led to a massive improvement, increasing the fraction of proven assertions by 29% in 1-shot scenarios and 38% in 5-shot scenarios. Furthermore, fine-tuned models reduced the generation of logically flawed assertions (CEX) by nearly half (48%) in certain configurations.
The Complexity of Scale
One surprising insight was that "bigger isn't always better." For example, LLaMa 3-70B actually saw a decrease in accuracy when moving from 1-shot to 5-shot learning. The increased variety of examples likely confused the model regarding the strict, rigid syntax required for SVA.
Discussion and Future Work
While AssertionLLM represents a significant step forward, challenges remain. Even with fine-tuning, models produced syntactically incorrect code up to 38% of the time. This suggests that LLMs struggle with the precise grammar of hardware description languages compared to natural language or common software languages like Python.
Future research directions include:
- Expanding Language Support: Moving beyond Verilog and SVA to include VHDL and SystemC.
- Auxiliary Artifacts: Feeding models more than just raw source code. Incorporating Control-Data Flow Graphs (CDFG) or Variable Dependency Graphs could help models "visualize" hardware data movement.
- Dataset Expansion: Building larger, more diverse datasets to help models learn the nuances and edge cases of hardware properties.
Conclusion
Are LLMs ready for practical assertion generation? For off-the-shelf models, the answer is no. However, the success of AssertionLLM proves that fine-tuning and formal verification loops can bridge the gap. By moving away from "black box" prompting and toward domain-specific architectures, the hardware industry can begin to automate one of its most tedious and critical tasks.
Summary updated