Are LLMs Ready for Practical Adoption for Assertion Generation?

Vaishnavi Pulavarthi, Deeksha Nandal, Soham Dan, Debjit Pal

TL;DR

This paper investigates whether Large Language Models (LLMs) are ready to automate the tedious task of generating hardware verification assertions. The authors introduce AssertionBench, a new benchmark for evaluating LLMs on Verilog designs, and AssertionLLM, a fine-tuned model that improves assertion accuracy by up to 38% over off-the-shelf models, though significant hurdles in syntax and semantic correctness remain.

What Problem Does This Paper Address?

In modern chip design (System-on-Chip or SoC), "assertions" are critical mathematical rules used to verify that a design behaves correctly and securely. For example, an assertion might state: "If a 'request' signal is high, a 'grant' signal must follow in the next cycle."

Currently, writing these assertions is a manual bottleneck that requires deep expertise and "human ingenuity." While automated tools exist, they often fail to scale to industrial designs or produce a flood of redundant, useless properties. With the rise of generative AI, researchers are asking: Can LLMs write these assertions for us? This paper seeks to quantify exactly how close (or far) we are from that reality.

Methods

The researchers took a two-pronged approach:

  1. Evaluation of COTS Models: They tested "Commercial-Off-The-Shelf" (COTS) models—including GPT-3.5, GPT-4o, CodeLLaMa 2, and LLaMa 3-70B—using In-Context Learning (ICL). This involves giving the model a few examples (1-shot or 5-shot) of Verilog code and its corresponding assertions before asking it to generate new ones.
  2. Fine-Tuning (AssertionLLM): Recognizing that general-purpose models struggle with hardware-specific logic, they created AssertionLLM. They fine-tuned CodeLLaMa 2 and LLaMa 3-70B on a specialized dataset of Verilog designs and formally verified assertions.
  3. Formal Verification Loop: Unlike many AI papers that just check if text "looks right," this study used the Cadence JasperGold formal verification engine. This allowed them to prove whether the AI-generated assertions were actually mathematically valid or if they triggered a "Counter-Example" (CEX), meaning the assertion was logically flawed.

Datasets

The authors introduced AssertionBench, a comprehensive suite of 100 Verilog designs sourced from OpenCores.

  • Diversity: It includes both combinational and sequential designs, ranging from simple arbiters to complex components like Floating Point Units (FPUs) and Ethernet controllers.
  • Scale: Design sizes vary from 10 to 1,150 lines of code.
  • Ground Truth: Each design is paired with a set of formally verified assertions (average of 4.8 per design) to serve as the gold standard for evaluation.

Key Insights and Ideas

  • Bigger isn't always better: Surprisingly, more sophisticated models didn't always perform better. For instance, LLaMa 3-70B actually lost accuracy when moved from 1-shot to 5-shot learning, likely because the increased variety of examples confused its understanding of the strict assertion syntax.
  • GPT-4o is the "COTS King": Among off-the-shelf models, GPT-4o was the most consistent, producing fewer syntax errors and more valid assertions than its peers.
  • The "Black Box" Limit: Treating an LLM as a black box and relying solely on prompt engineering (ICL) has a hard ceiling. The models often fail to grasp the underlying hardware semantics from raw source code alone.
  • Fine-tuning is the path forward: Fine-tuning significantly boosted the models' ability to generate "proven" (valid) assertions, showing that domain-specific training is non-negotiable for hardware engineering tasks.

Key Results

  • COTS Failure Rates: Off-the-shelf models are not "prime-time ready." On average, they achieved no more than 44% accuracy. Up to 63% of their assertions were semantically incorrect (failed formal verification), and 33% were syntactically broken.
  • AssertionLLM Gains: Fine-tuning CodeLLaMa 2 led to a massive improvement, increasing the fraction of proven assertions by 29% (1-shot) and 38% (5-shot).
  • Error Reduction: Fine-tuned models reduced the generation of counter-examples (logically wrong assertions) by nearly half (48%) in some configurations.
  • The Syntax Persistent: Even with fine-tuning, models still produced syntactically incorrect code up to 38% of the time, highlighting the difficulty LLMs have with the rigid grammar of SystemVerilog Assertions (SVA).

Limitations and Future Work

  • Language Constraints: The study focused exclusively on Verilog and SVA. Future work needs to expand to VHDL and SystemC.
  • Beyond Raw Code: The authors suggest that LLMs need more than just source code to succeed. Future models should be fed "auxiliary artifacts" like Control-Data Flow Graphs (CDFG) or Variable Dependency Graphs to help the AI "visualize" how data moves through the hardware.
  • Dataset Breadth: To solve the syntax error problem, a much larger and more diverse dataset of hardware properties is required to help models learn the "edge cases" of hardware description languages.

Summary updated

Abstract

Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification, i.e., detection and diagnosis of corner-case design bugs, is critically dependent on the quality of the assertions. With the onset of generative AI such as Transformers and Large-Language Models (LLMs), there has been a renewed interest in developing novel, effective, and scalable techniques of generating functional and security assertions from design source code. While there have been recent works that use commercial-of-the-shelf (COTS) LLMs for assertion generation, there is no comprehensive study in quantifying the effectiveness of LLMs in generating syntactically and semantically correct assertions. In this paper, we first discuss AssertionBench from our prior work, a comprehensive set of designs and assertions to quantify the goodness of a broad spectrum of COTS LLMs for the task of assertion generations from hardware design source code. Our key insight was that COTS LLMs are not yet ready for prime-time adoption for assertion generation as they generate a considerable fraction of syntactically and semantically incorrect assertions. Motivated by the insight, we propose AssertionLLM, a first of its kind LLM model, specifically fine-tuned for assertion generation. Our initial experimental results show that AssertionLLM considerably improves the semantic and syntactic correctness of the generated assertions over COTS LLMs.
BibTeX
@inproceedings{pulavarthi2025are,
  author = {Vaishnavi Pulavarthi and Deeksha Nandal and Soham Dan and Debjit Pal},
  title = {{Are LLMs Ready for Practical Adoption for Assertion Generation?}},
  booktitle = {DATE},
  year = {2025},
  doi = {10.23919/date64628.2025.10992817},
}

Loading related content...

Figure