TL;DR
FVRULELEARNER is a novel framework that helps Large Language Models (LLMs) generate accurate SystemVerilog Assertions (SVAs) from natural language specifications by learning from its own mistakes. By breaking down errors into structured "Operator Reasoning Trees" (OP-TREE), it extracts precise, symbol-level rules that improve the functional correctness of generated hardware verification code by an average of 31.17% over state-of-the-art baselines.
What Problem Does This Paper Address?
Formal verification (FV) is a critical phase in hardware design that mathematically proves whether a circuit behaves correctly under all conditions. To do this, engineers write SystemVerilog Assertions (SVAs) to define the design's intended behavior. However, manually writing SVAs from natural language specifications is tedious and error-prone.
While LLMs are great at writing standard software code, they struggle significantly with SVAs. The core issue isn't syntax; it's semantics. Specifically, LLMs struggle with temporal operators—the logic governing when events should happen (e.g., "same-cycle" vs. "next-cycle" implications).
As shown in the paper's analysis, temporal operators account for over 80% of all functional failures, with basic implication operators (|-> vs |=>) alone causing up to 64% of errors.
Methods
Instead of relying on abstract, high-level advice or expensive model fine-tuning, FVRULELEARNER introduces a structured, two-phase rule-learning paradigm.
1. Training Phase: Building the Op-Tree
When the LLM generates an incorrect SVA during training, the framework uses the correct "golden" SVA as an oracle to diagnose the mistake. It constructs an Operator Reasoning Tree (OP-TREE) through three layers:
- Contextual Diagnosis: Analyzes signal relationships to hypothesize why the failure occurred.
- Theoretical Grounding: Validates the hypothesis against formal axioms (e.g., distinguishing between overlapping and non-overlapping implications).
- Rule Generation: Synthesizes this knowledge into a concrete Op-Rule.
Crucially, a rule is only saved to the database if it successfully guides the LLM to generate a functionally perfect SVA in a self-reflection loop.
2. Testing Phase: Hybrid Retrieval and Rule Adaptation
For an unseen natural language prompt, the framework:
- Generates an initial, unguided SVA.
- Retrieves relevant reasoning traces from its database using a hybrid score that balances linguistic similarity (standard RAG) with an Operator Alignment Score (using a soft Jaccard similarity to match intended operators).
- Applies Signal Abstraction to mask specific variable names from the training data with generic placeholders (e.g.,
<signal>), allowing the rule to generalize to entirely new hardware designs.
Datasets
The authors evaluated their framework across three distinct datasets:
- NL2SVA-HUMAN: Human-curated natural language to SVA pairs.
- NL2SVA-MACHINE: Machine-generated specifications.
- NL2SVA-OPENCORE: A brand-new dataset introduced in this paper, consisting of 1,000 NL-SVA pairs mined from real-world, open-source hardware designs.
Each dataset was split into 80% training and 20% testing subsets.
Key Insights and Ideas
- Execution Over Interpretation: General rules (e.g., "align assertion syntax with intended temporal logic") fail because LLMs cannot reliably translate abstract guidance into precise syntax. Deterministic, symbol-level directives ("Replace
|->with|=>when...") force correct execution. - Rule Saturation (Data Efficiency): The authors discovered that functional correctness plateaus at a mere 48% training data ratio. This means performance depends on the diversity of operator rules encountered during training, rather than the sheer volume of data, making the framework highly data-efficient.
- Targeted Debugging: By injecting formal definitions directly into the context window at inference time, the framework effectively "debugs" the LLM's temporal logic before the final code is generated.
Key Results
The framework was tested using top-tier commercial models, including GPT-4o, Claude 4.5 Sonnet, and o3-mini.
- Functional Correctness: FVRULELEARNER outperformed state-of-the-art baselines by an average of 31.17% in strict functional correctness.
- Syntax Accuracy: It achieved an average of 98.56% syntactic correctness.
- Error Reduction: Across all datasets, it reduced total functional mismatches by an average of 70.33%.
- Temporal Mastery: It achieved an 80.6% reduction in temporal implication errors and an 85.0% reduction in temporal delay errors, proving it successfully resolved the LLM's primary blind spots.
Limitations and Future Work
While FVRULELEARNER dramatically mitigates operator-level errors, it is still bounded by the underlying reasoning capabilities of the base LLM. The remaining failures typically stem from highly complex, globally coupled dependencies or fundamentally ambiguous natural language descriptions.
Moving forward, the authors plan to integrate FVRULELEARNER into industrial verification pipelines and explore combining this retrieval-based rule approach with domain-specific fine-tuning.
Summary updated