TL;DR
This paper investigates whether Large Language Models (LLMs) are ready to automate the tedious task of generating hardware verification assertions. The authors introduce AssertionBench, a new benchmark for evaluating LLMs on Verilog designs, and AssertionLLM, a fine-tuned model that improves assertion accuracy by up to 38% over off-the-shelf models, though significant hurdles in syntax and semantic correctness remain.
What Problem Does This Paper Address?
In modern chip design (System-on-Chip or SoC), "assertions" are critical mathematical rules used to verify that a design behaves correctly and securely. For example, an assertion might state: "If a 'request' signal is high, a 'grant' signal must follow in the next cycle."
Currently, writing these assertions is a manual bottleneck that requires deep expertise and "human ingenuity." While automated tools exist, they often fail to scale to industrial designs or produce a flood of redundant, useless properties. With the rise of generative AI, researchers are asking: Can LLMs write these assertions for us? This paper seeks to quantify exactly how close (or far) we are from that reality.
Methods
The researchers took a two-pronged approach:
- Evaluation of COTS Models: They tested "Commercial-Off-The-Shelf" (COTS) models—including GPT-3.5, GPT-4o, CodeLLaMa 2, and LLaMa 3-70B—using In-Context Learning (ICL). This involves giving the model a few examples (1-shot or 5-shot) of Verilog code and its corresponding assertions before asking it to generate new ones.
- Fine-Tuning (AssertionLLM): Recognizing that general-purpose models struggle with hardware-specific logic, they created AssertionLLM. They fine-tuned CodeLLaMa 2 and LLaMa 3-70B on a specialized dataset of Verilog designs and formally verified assertions.
- Formal Verification Loop: Unlike many AI papers that just check if text "looks right," this study used the Cadence JasperGold formal verification engine. This allowed them to prove whether the AI-generated assertions were actually mathematically valid or if they triggered a "Counter-Example" (CEX), meaning the assertion was logically flawed.
Datasets
The authors introduced AssertionBench, a comprehensive suite of 100 Verilog designs sourced from OpenCores.
- Diversity: It includes both combinational and sequential designs, ranging from simple arbiters to complex components like Floating Point Units (FPUs) and Ethernet controllers.
- Scale: Design sizes vary from 10 to 1,150 lines of code.
- Ground Truth: Each design is paired with a set of formally verified assertions (average of 4.8 per design) to serve as the gold standard for evaluation.
Key Insights and Ideas
- Bigger isn't always better: Surprisingly, more sophisticated models didn't always perform better. For instance, LLaMa 3-70B actually lost accuracy when moved from 1-shot to 5-shot learning, likely because the increased variety of examples confused its understanding of the strict assertion syntax.
- GPT-4o is the "COTS King": Among off-the-shelf models, GPT-4o was the most consistent, producing fewer syntax errors and more valid assertions than its peers.
- The "Black Box" Limit: Treating an LLM as a black box and relying solely on prompt engineering (ICL) has a hard ceiling. The models often fail to grasp the underlying hardware semantics from raw source code alone.
- Fine-tuning is the path forward: Fine-tuning significantly boosted the models' ability to generate "proven" (valid) assertions, showing that domain-specific training is non-negotiable for hardware engineering tasks.
Key Results
- COTS Failure Rates: Off-the-shelf models are not "prime-time ready." On average, they achieved no more than 44% accuracy. Up to 63% of their assertions were semantically incorrect (failed formal verification), and 33% were syntactically broken.
- AssertionLLM Gains: Fine-tuning CodeLLaMa 2 led to a massive improvement, increasing the fraction of proven assertions by 29% (1-shot) and 38% (5-shot).
- Error Reduction: Fine-tuned models reduced the generation of counter-examples (logically wrong assertions) by nearly half (48%) in some configurations.
- The Syntax Persistent: Even with fine-tuning, models still produced syntactically incorrect code up to 38% of the time, highlighting the difficulty LLMs have with the rigid grammar of SystemVerilog Assertions (SVA).
Limitations and Future Work
- Language Constraints: The study focused exclusively on Verilog and SVA. Future work needs to expand to VHDL and SystemC.
- Beyond Raw Code: The authors suggest that LLMs need more than just source code to succeed. Future models should be fed "auxiliary artifacts" like Control-Data Flow Graphs (CDFG) or Variable Dependency Graphs to help the AI "visualize" how data moves through the hardware.
- Dataset Breadth: To solve the syntax error problem, a much larger and more diverse dataset of hardware properties is required to help models learn the "edge cases" of hardware description languages.
Summary updated