FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification

Lily Jiaxin Wan, Chia-Tung Ho, Yunsheng Bai, Cunxi Yu, Deming Chen, Haoxing Ren

TL;DR

FVRULELEARNER is a novel framework that helps Large Language Models (LLMs) generate accurate SystemVerilog Assertions (SVAs) from natural language specifications by learning from its own mistakes. By breaking down errors into structured "Operator Reasoning Trees" (OP-TREE), it extracts precise, symbol-level rules that improve the functional correctness of generated hardware verification code by an average of 31.17% over state-of-the-art baselines.

What Problem Does This Paper Address?

Formal verification (FV) is a critical phase in hardware design that mathematically proves whether a circuit behaves correctly under all conditions. To do this, engineers write SystemVerilog Assertions (SVAs) to define the design's intended behavior. However, manually writing SVAs from natural language specifications is tedious and error-prone.

While LLMs are great at writing standard software code, they struggle significantly with SVAs. The core issue isn't syntax; it's semantics. Specifically, LLMs struggle with temporal operators—the logic governing when events should happen (e.g., "same-cycle" vs. "next-cycle" implications).

Operator-level functional mismatch analysis across datasets shows that temporal operators dominate errors.

As shown in the paper's analysis, temporal operators account for over 80% of all functional failures, with basic implication operators (|-> vs |=>) alone causing up to 64% of errors.

Methods

Instead of relying on abstract, high-level advice or expensive model fine-tuning, FVRULELEARNER introduces a structured, two-phase rule-learning paradigm.

The flow overview of FVRULELEARNER showing the Training phase (a), Testing phase (b), and an illustration of the Op-Tree structure (c).

1. Training Phase: Building the Op-Tree

When the LLM generates an incorrect SVA during training, the framework uses the correct "golden" SVA as an oracle to diagnose the mistake. It constructs an Operator Reasoning Tree (OP-TREE) through three layers:

  • Contextual Diagnosis: Analyzes signal relationships to hypothesize why the failure occurred.
  • Theoretical Grounding: Validates the hypothesis against formal axioms (e.g., distinguishing between overlapping and non-overlapping implications).
  • Rule Generation: Synthesizes this knowledge into a concrete Op-Rule.

Crucially, a rule is only saved to the database if it successfully guides the LLM to generate a functionally perfect SVA in a self-reflection loop.

2. Testing Phase: Hybrid Retrieval and Rule Adaptation

For an unseen natural language prompt, the framework:

  1. Generates an initial, unguided SVA.
  2. Retrieves relevant reasoning traces from its database using a hybrid score that balances linguistic similarity (standard RAG) with an Operator Alignment Score (using a soft Jaccard similarity to match intended operators).
  3. Applies Signal Abstraction to mask specific variable names from the training data with generic placeholders (e.g., <signal>), allowing the rule to generalize to entirely new hardware designs.
Comparison showing how specific Op-Rules succeed where abstract General Rules fail.

Datasets

The authors evaluated their framework across three distinct datasets:

  • NL2SVA-HUMAN: Human-curated natural language to SVA pairs.
  • NL2SVA-MACHINE: Machine-generated specifications.
  • NL2SVA-OPENCORE: A brand-new dataset introduced in this paper, consisting of 1,000 NL-SVA pairs mined from real-world, open-source hardware designs.

Each dataset was split into 80% training and 20% testing subsets.

Key Insights and Ideas

  • Execution Over Interpretation: General rules (e.g., "align assertion syntax with intended temporal logic") fail because LLMs cannot reliably translate abstract guidance into precise syntax. Deterministic, symbol-level directives ("Replace |-> with |=> when...") force correct execution.
  • Rule Saturation (Data Efficiency): The authors discovered that functional correctness plateaus at a mere 48% training data ratio. This means performance depends on the diversity of operator rules encountered during training, rather than the sheer volume of data, making the framework highly data-efficient.
  • Targeted Debugging: By injecting formal definitions directly into the context window at inference time, the framework effectively "debugs" the LLM's temporal logic before the final code is generated.

Key Results

The framework was tested using top-tier commercial models, including GPT-4o, Claude 4.5 Sonnet, and o3-mini.

  • Functional Correctness: FVRULELEARNER outperformed state-of-the-art baselines by an average of 31.17% in strict functional correctness.
  • Syntax Accuracy: It achieved an average of 98.56% syntactic correctness.
  • Error Reduction: Across all datasets, it reduced total functional mismatches by an average of 70.33%.
  • Temporal Mastery: It achieved an 80.6% reduction in temporal implication errors and an 85.0% reduction in temporal delay errors, proving it successfully resolved the LLM's primary blind spots.

Limitations and Future Work

While FVRULELEARNER dramatically mitigates operator-level errors, it is still bounded by the underlying reasoning capabilities of the base LLM. The remaining failures typically stem from highly complex, globally coupled dependencies or fundamentally ambiguous natural language descriptions.

Moving forward, the authors plan to integrate FVRULELEARNER into industrial verification pipelines and explore combining this retrieval-based rule approach with domain-specific fine-tuning.

Summary updated

Abstract

The remarkable reasoning and code generation capabilities of large language models (LLMs) have recently motivated increasing interest in automating formal verification (FV), a process that ensures hardware correctness through mathematically precise assertions but remains highly labor-intensive, particularly through the translation of natural language into SystemVerilog Assertions (NL-to-SVA). However, LLMs still struggle with SVA generation due to limited training data and the intrinsic complexity of FV operators. Consequently, a more efficient and robust methodology for ensuring correct SVA operator selection is essential for producing functionally correct assertions. To address these challenges, we introduce FVRuleLearner, an Operator-Level Rule (Op-Rule) learning framework built on a novel Operator Reasoning Tree (Op-Tree), which models SVA generation as structured, interpretable reasoning. FVRuleLearner operates in two complementary phases: (1) Training: it constructs Op-Tree that decomposes NL-to-SVA alignment into fine-grained, operator-aware questions, combining reasoning paths that lead to correct assertions; and (2) Testing: it performs operator-aligned retrieval to fetch relevant reasoning traces from the learned Op-Tree and generate new rules for unseen specifications. In the comprehensive studies, the proposed FVRuleLearner outperforms the state-of-the-art baseline by 3.95% in syntax correctness and by 31.17% in functional correctness on average. Moreover, FVRuleLearner successfully reduces an average of 70.33% of SVA functional failures across diverse operator categories through a functional taxonomy analysis, showing the effectiveness of applying learned Op-Tree to the Op-Rule generations for unseen NL-to-SVA tasks. These results establish FVRuleLearner as a new paradigm for domain-specific reasoning and rule learning in formal verification.
BibTeX
@inproceedings{wan2026fvrulelearner,
  author = {Lily Jiaxin Wan and Chia-Tung Ho and Yunsheng Bai and Cunxi Yu and Deming Chen and Haoxing Ren},
  title = {{FVRuleLearner: Operator-Level Reasoning Tree (Op-Tree)-Based Rules Learning for Formal Verification}},
  booktitle = {VTS},
  year = {2026},
  doi = {10.1109/vts69484.2026.11563220},
}
Figure