Are LLMs Ready for Practical Adoption for Assertion Generation?

Vaishnavi Pulavarthi, Deeksha Nandal, Soham Dan, Debjit Pal

Abstract

Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification—specifically the detection and diagnosis of corner-case design bugs—is critically dependent on the quality of these assertions. With the rise of generative AI, there is renewed interest in using Large Language Models (LLMs) to generate functional and security assertions from design source code. This paper introduces AssertionBench, a comprehensive set of designs and assertions used to quantify the effectiveness of commercial-off-the-shelf (COTS) LLMs. Our findings suggest that COTS LLMs are not yet ready for practical adoption due to high rates of syntactically and semantically incorrect outputs. To address this, we propose AssertionLLM, a first-of-its-kind model fine-tuned specifically for assertion generation, which demonstrates significant improvements in accuracy over general-purpose models.

Introduction

In modern System-on-Chip (SoC) design, assertions serve as the mathematical rules that ensure a design behaves correctly and securely. For instance, a simple assertion might dictate that a 'grant' signal must follow a 'request' signal within a specific timeframe. Despite their importance, writing high-quality assertions remains a manual bottleneck requiring deep domain expertise. While automated tools exist, they often struggle to scale or produce redundant properties.

As Large Language Models (LLMs) transform software engineering, the hardware community is exploring whether these models can automate assertion generation. This study evaluates the current state of the art and proposes a specialized path forward through domain-specific fine-tuning.

AssertionBench: A New Standard for Evaluation

To rigorously test LLM capabilities, we developed AssertionBench, a suite of 100 Verilog designs sourced from OpenCores. The benchmark is characterized by:

  • Diversity: It includes combinational and sequential designs, ranging from simple arbiters to complex components like Floating Point Units (FPUs) and Ethernet controllers.
  • Scale: Design sizes vary from 10 to 1,150 lines of code.
  • Ground Truth: Each design is paired with an average of 4.8 formally verified assertions to serve as the gold standard for evaluation.

Methodology

Our research followed a two-pronged approach to evaluate and improve assertion generation:

1. Evaluation of COTS Models

We tested several general-purpose models, including GPT-3.5, GPT-4o, CodeLLaMa 2, and LLaMa 3-70B. We utilized In-Context Learning (ICL), providing the models with 1-shot or 5-shot examples of Verilog code and corresponding SystemVerilog Assertions (SVA) to guide their output.

2. Development of AssertionLLM

Recognizing the limitations of general-purpose models, we created AssertionLLM. We fine-tuned CodeLLaMa 2 and LLaMa 3-70B on a specialized dataset of Verilog designs and verified assertions to better capture hardware-specific logic.

3. Formal Verification Loop

To move beyond simple text-matching metrics, we employed the Cadence JasperGold formal verification engine. This allowed us to determine if AI-generated assertions were mathematically valid or if they triggered a Counter-Example (CEX), indicating a logical flaw.

Key Insights and Results

The Limitations of COTS Models

Off-the-shelf models are currently unsuitable for production-level hardware verification. On average, COTS models achieved no more than 44% accuracy. A staggering 63% of their assertions were semantically incorrect, and 33% were syntactically broken. Interestingly, GPT-4o emerged as the most consistent COTS model, producing the fewest syntax errors.

The Impact of Fine-Tuning

AssertionLLM demonstrated that domain-specific training is essential. Fine-tuning CodeLLaMa 2 led to a massive improvement, increasing the fraction of proven assertions by 29% in 1-shot scenarios and 38% in 5-shot scenarios. Furthermore, fine-tuned models reduced the generation of logically flawed assertions (CEX) by nearly half (48%) in certain configurations.

The Complexity of Scale

One surprising insight was that "bigger isn't always better." For example, LLaMa 3-70B actually saw a decrease in accuracy when moving from 1-shot to 5-shot learning. The increased variety of examples likely confused the model regarding the strict, rigid syntax required for SVA.

Discussion and Future Work

While AssertionLLM represents a significant step forward, challenges remain. Even with fine-tuning, models produced syntactically incorrect code up to 38% of the time. This suggests that LLMs struggle with the precise grammar of hardware description languages compared to natural language or common software languages like Python.

Future research directions include:

  • Expanding Language Support: Moving beyond Verilog and SVA to include VHDL and SystemC.
  • Auxiliary Artifacts: Feeding models more than just raw source code. Incorporating Control-Data Flow Graphs (CDFG) or Variable Dependency Graphs could help models "visualize" hardware data movement.
  • Dataset Expansion: Building larger, more diverse datasets to help models learn the nuances and edge cases of hardware properties.

Conclusion

Are LLMs ready for practical assertion generation? For off-the-shelf models, the answer is no. However, the success of AssertionLLM proves that fine-tuning and formal verification loops can bridge the gap. By moving away from "black box" prompting and toward domain-specific architectures, the hardware industry can begin to automate one of its most tedious and critical tasks.

Summary updated

Abstract

Assertions have been the de facto collateral for simulation-based and formal verification of hardware designs for over a decade. The quality of hardware verification, i.e., detection and diagnosis of corner-case design bugs, is critically dependent on the quality of the assertions. With the onset of generative AI such as Transformers and Large-Language Models (LLMs), there has been a renewed interest in developing novel, effective, and scalable techniques of generating functional and security assertions from design source code. While there have been recent works that use commercial-of-the-shelf (COTS) LLMs for assertion generation, there is no comprehensive study in quantifying the effectiveness of LLMs in generating syntactically and semantically correct assertions. In this paper, we first discuss AssertionBench from our prior work, a comprehensive set of designs and assertions to quantify the goodness of a broad spectrum of COTS LLMs for the task of assertion generations from hardware design source code. Our key insight was that COTS LLMs are not yet ready for prime-time adoption for assertion generation as they generate a considerable fraction of syntactically and semantically incorrect assertions. Motivated by the insight, we propose AssertionLLM, a first of its kind LLM model, specifically fine-tuned for assertion generation. Our initial experimental results show that AssertionLLM considerably improves the semantic and syntactic correctness of the generated assertions over COTS LLMs.
BibTeX
@inproceedings{pulavarthi2025are,
  author = {Vaishnavi Pulavarthi and Deeksha Nandal and Soham Dan and Debjit Pal},
  title = {{Are LLMs Ready for Practical Adoption for Assertion Generation?}},
  booktitle = {DATE},
  year = {2025},
  doi = {10.23919/date64628.2025.10992817},
}
Figure