VerusEval: Evaluating LLM-Generated Verus Specifications Beyond Verifier Acceptance

Ding Wang, Chao Jiang, Dugang Liu, Zhiwu Xu, Cheng Wen, Zhong Ming, Yuwei Liu, Xinyi Wan, Lin HuangarXiv 2026 · arXiv

TL;DR

Verifier acceptance is a deceptive metric for formal specifications: just because a contract passes the verifier doesn't mean it's correct or useful. This paper introduces VerusEval, a multi-dimensional framework that reveals over half of "verified" LLM-generated contracts contain semantic flaws, and provides a roadmap for more robust evaluation and benchmark maintenance.

What Problem Does This Paper Address?

In the world of formal verification, we use tools like Verus to prove that Rust code satisfies a mathematical specification (a "contract"). Large Language Models (LLMs) are increasingly used to automate this tedious process by generating both the code and the contract.

The standard measure of success has been "verifier acceptance"—if the tool says the code satisfies the contract, we win. However, the authors identify a massive "vacuity" problem: a contract can be technically "accepted" even if it says nothing useful (e.g., a postcondition that is always true) or if it omits critical requirements. Current evaluation methods fail to distinguish between a rigorous, correct specification and a "loose" one that passes only because it's too vague.

Methods

The authors propose VerusEval, which evaluates generated contracts across six dimensions to capture their true semantic quality:

  1. Verifier Validity: Does it pass parsing, type-checking, and the final proof?
  2. Semantic Triviality: Is the contract vacuous? (e.g., an impossible precondition or a meaningless postcondition).
  3. Specification Correctness: How does the generated contract compare logically to a "gold standard" reference? The authors natively encode soundness and completeness as Verus proof obligations.
  4. Behavior Correctness: Does the contract admit correct I/O pairs and reject incorrect ones?
  5. Intent Judgement: An "LLM-as-a-judge" assessment of whether the contract matches the implementation's intent.
  6. Text Similarity: Standard NLP metrics like BLEU and ROUGE.
Fig. 2. Overview of VerusEval.

A key innovation is the use of Failure Directions. By checking implications like $P_0 \Rightarrow P$ (Stronger Precondition) or $Q \Rightarrow Q_0$ (Weaker Postcondition), the framework can diagnose exactly how a contract differs from the reference—whether it's too restrictive or too permissive.

Datasets

The study is massive in scale, evaluating 13,716 artifacts across:

  • 762 tasks from five existing benchmarks (including VerusBench and HumanEval-Verus).
  • 4 workflows (AlphaVerus, AutoVerus, StarVerus, VeruSAGE).
  • 5 LLMs (GPT-4o, Llama 4 Maverick, DeepSeek-Chat/Reasoner, Qwen-Coder).
  • Validated I/O suites containing 11,641 cases (correct pairs, wrong outputs, and invalid inputs).

Key Insights and Ideas

  • Acceptance $\neq$ Quality: Higher verifier acceptance rates do not consistently lead to more correct specifications. Some workflows appear "better" simply because they generate looser, easier-to-verify contracts.
  • The "Blind Spot" of I/O: While I/O testing is a great first filter, it's weak evidence for full equivalence. 23.6% of accepted artifacts failed I/O, but of those that passed I/O, only 63.2% were actually proved equivalent to the reference.
  • Auditing the Oracle: If an LLM generates a contract that is "stronger" than the reference, it might not be a hallucination—the reference itself might be defective. The authors found 36 defects in 61 reviewed reference contracts.
  • Directional Diagnosis: By looking at which specific logical obligation failed (e.g., $L_1$ vs $L_4$), researchers can prioritize whether to fix the LLM's output or audit the benchmark's reference.

Key Results

  • The Quality Gap: Only 42.3% of verifier-accepted artifacts passed all three semantic checks (triviality, I/O, and contract comparison) without a flag.
  • Reference Defects: Screening identified 36 candidate defects in reference contracts. In one case study, repairing a single missing guarantee in a reference contract made 10 out of 11 previously "failing" LLM artifacts equivalent.
  • Prompting Matters: Few-shot prompting significantly raised acceptance and quality across all models, but much of the I/O gain was simply due to more artifacts becoming "type-valid" (compilable) rather than "smarter."
Table 1. Core metrics.

Limitations and Future Work

  • Reference Dependence: The framework's strongest checks still rely on a reference contract, which the authors proved can be fallible.
  • Proof Automation Limits: A "rejected" obligation might sometimes stem from the SMT solver timing out rather than a true logical difference.
  • Future Directions: The authors suggest using I/O counterexamples to automatically "repair" weaker generated contracts and using cross-model agreement to automatically flag potentially defective benchmark references.

Summary updated

Abstract

Large language models (LLMs) are increasingly used to generate Verus contracts for Rust programs together with the proofs that establish them. Yet verifier acceptance, the common measure of success, shows only that the implementation satisfies the generated contract, so it leaves open whether the contract captures the intended behavior. Assessing specification generation therefore requires an evaluation methodology, not a single metric. We present VerusEval, a framework that evaluates generated contracts along six dimensions, including Verus-native checks for semantic triviality, validated input-output (I/O) suites, and contract equivalence, and that relates them so that the strength and blind spots of each become measurable. Evaluating 13,716 artifacts from four workflows, five LLMs, and two prompting settings on 762 tasks, we find that acceptance overstates quality: only 42.3\% of the accepted artifacts complete all three semantic checks without a flag, and across workflows and models, higher acceptance does not consistently come with more correct contracts. I/O evaluation rejects 23.6\% of the accepted artifacts, yet only 63.2\% of those that pass it are proved equivalent. Of the accepted artifacts flagged by contract comparison, 84.1\% have a rejected completeness obligation, and the direction of a failure prioritizes inspection of the generated contract or the reference; it does not by itself establish which artifact is defective. Screening and review further identify 36 candidate defects among 61 selected reference contracts, and repairing one of them makes 10 of its 11 failing artifacts equivalent. Based on these findings, we derive four actionable lessons for evaluation, generation, and benchmark maintenance.
BibTeX
@misc{wang2026veruseval,
  author = {Ding Wang and Chao Jiang and Dugang Liu and Zhiwu Xu and Cheng Wen and Zhong Ming and Yuwei Liu and Xinyi Wan and Lin Huang},
  title = {{VerusEval: Evaluating LLM-Generated Verus Specifications Beyond Verifier Acceptance}},
  year = {2026},
  eprint = {2610.26464},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.26464},
}
Figure