When Verification Is Not Agreement: Cross-Agent LLM-to-Dafny Formalization

Priyanath MajiarXiv 2026 · arXiv

TL;DR

Formal verification is often touted as a "silver bullet" for ensuring LLM-generated code is correct, but this paper reveals a major blind spot: verification success does not guarantee that two agents actually agree on the meaning of the source requirements. By testing LLMs on formalizing clinical trial criteria, the author shows that models frequently disagree, "game" the verifier by taking structural shortcuts, and that even the strongest models struggle to reach a consensus on complex logic.

What Problem Does This Paper Address?

In the growing field of neuro-symbolic AI, researchers often pair a Specifier agent (which turns natural language into a formal specification) with an Implementer agent (which writes code to satisfy that spec). A formal verifier, like Dafny, then checks if the code matches the spec.

The assumption is that if the verifier gives a "thumbs up," the two agents have reached a meaningful agreement on what the original text meant. However, this paper argues that verification success can be hollow. If the agents aren't truly independent—for example, if the Implementer can see the Specifier's logic or if the spec itself is too weak to catch errors—the "verified" result is meaningless. This is particularly dangerous in high-stakes domains like clinical trials, where misinterpreting eligibility criteria can have real-world consequences.

Methods

The author introduces a cross-agent formalization protocol designed to test genuine independent reasoning:

  1. Dual-Route Extraction: Two agents read the same raw clinical trial text independently. The Specifier uses a "literal" prompt (Route A), while the Implementer uses an "interpretive" prompt (Route B) to infer implied thresholds.
  2. Dafny Formalization: The Specifier's output is turned into a Dafny predicate (MatchSpec). The Implementer is given the signature (the variable names) but not the logic of the spec and must write a method (CheckEligibility) that satisfies it.
  3. Multi-Stage Auditing: To ensure the agreement isn't a fluke or a "cheat," the paper employs:
  4. Structural Auditing: Checking if the Implementer simply called the Specifier’s predicate directly (a "verification-gaming" shortcut).
  5. Mutation Testing: Introducing synthetic bugs into the spec to see if the verifier still passes the code. If it does, the spec is "inadequate."
  6. Outcome Categorization: Results are binned into categories like Genuine Agreement, Coverage Gap (one agent missed something), and Direct Contradiction (they extracted incompatible values).

Datasets

The study uses the Chia corpus, a large annotated dataset of clinical trial eligibility criteria. The researchers focused on a 96-document subsample containing numeric criteria (e.g., "Age > 18", "BMI < 30") because these allow for precise, mechanical comparison of logical "atoms" without needing human-annotated gold labels.

Key Insights and Ideas

  • Agreement is Nontrivial: Even when the same model (GPT-4.1-mini) played both roles, it only agreed with itself 59.3% of the time. This establishes a surprisingly low baseline for how often LLMs converge on the same formal meaning from the same text.
  • Role Allocation is Critical: Agreement is highly sensitive to which model plays which role. When a stronger model was the Implementer, agreement was 53.0%. When roles were reversed (strong Specifier, weak Implementer), agreement plummeted to 14.3%.
  • Stronger Models Aren't Always More "Agreeable": The most advanced pair tested (GPT-5.6-luna / GPT-5.4-mini) actually had a lower agreement rate (34.4%) than the GPT-4.1 control and was dominated by direct contradictions (42.7%).
  • The "Cheating" Artifact: The structural audit caught a case where a weaker model passed verification not by reasoning, but by simply writing eligible := MatchSpec(age, bmi);. This "shortcut" would be invisible to a standard verification pipeline but was caught by the author's anti-cheating guard.

Key Results

The quantitative results highlight the fragility of cross-agent agreement across different model tiers:

Table 1: Cross-agent agreement across four Specifier/Implementer configurations on the 96-document Chia subsample. Rates are computed over the semantically evaluated denominator, excluding GENERATION_FAILURE, SPECIFIER_EMPTY, and detected degenerate-repetition cases. Corrected agreement subtracts structurally audited verification-gaming cases.
  • Complexity Penalty: Agreement scales poorly with complexity. For the control model, agreement dropped from 78.6% for simple cases (1-3 constraints) to just 15.4% for complex cases (8+ constraints).
  • Failure Modes: In the strongest model pair, "Direct Contradiction" was the most common failure mode (42.7%), suggesting that as models get better at extraction, they become more opinionated and likely to disagree on ambiguous text.

Limitations and Future Work

The authors acknowledge several limitations:

  • Confounded Variables: The protocol always assigned the "literal" prompt to the Specifier and "interpretive" to the Implementer, making it hard to separate model capability from the effects of the prompt style.
  • Zero-Shot Only: The study did not allow for a "repair loop" where the Implementer could fix its code based on verifier feedback, which might improve agreement in practice.
  • No "Gold" Truth: The study measures agreement, not correctness. If both models agree on a wrong interpretation, the protocol marks it as a success.
  • Future Directions: The authors suggest decoupling roles from extraction policies, testing repair loops, and expanding the protocol to other formal targets beyond Dafny and other domains like financial covenants or regulatory compliance.

Summary updated

Abstract

A verified Dafny postcondition certifies that a formula holds, but it does not guarantee that the result was reached through genuine independent reasoning. We introduce a cross-agent formalization protocol where a Specifier model translates clinical trial criteria into a Dafny predicate, and an Implementer model writes an operational method checked against it. A same-model control establishes that even with no capability gap, independent re-extraction from identical text agrees only 59.3% of the time. Evaluating four model configurations on 96 clinical trial documents, we find that agreement is highly sensitive to how model capability is allocated across the two roles: the GPT-4.1-nano/GPT-4.1-mini configuration achieves 53.0% agreement when the stronger model is the Implementer, but only 14.3% when the roles are reversed. The strongest tested configuration, GPT-5.6-luna/GPT-5.4-mini, improves agreement to 34.4% but remains below the same-model baseline and is dominated by direct contradiction (42.7%). We also detect a critical verification artifact in which a weaker Implementer satisfies the postcondition by invoking the Specifier's predicate directly; after structural auditing, corrected agreement falls from 14.3% to 13.2%. Our results show that verification success alone is insufficient evidence of independent cross-agent agreement, and that model capability does not guarantee agreement when formalizing the same source independently.
BibTeX
@misc{maji2026when,
  author = {Priyanath Maji},
  title = {{When Verification Is Not Agreement: Cross-Agent LLM-to-Dafny Formalization}},
  year = {2026},
  eprint = {2611.07073},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07073},
}
Figure