Measuring Where Intent Gets Lost in Agentic Smart Contract Synthesis

Abhinav Goel, Agostino Capponi, Alfio Gliozzo, Chaitya ShaharXiv 2026 · arXiv

TL;DR

This paper demonstrates that for agentic code generation, gathering missing requirements before writing code is far more effective than trying to audit or repair code after the fact. By introducing a rigorous evaluation framework based on exact transaction replay and "elicitation-first" pipelines, the authors show that a few well-chosen questions can boost smart contract fidelity from 39.3% to 72.9%.

What Problem Does This Paper Address?

Large Language Model (LLM) agents are increasingly used to synthesize code from natural language. However, "code that compiles" is a low bar; the real challenge is "code that does what the user intended." This is especially critical for smart contracts, which manage high-value assets on blockchains and cannot be patched once deployed.

Current evaluation methods often rely on LLM-as-a-judge or static analysis, but these are "unvalidated instruments"—they often fail to catch subtle logic errors and can be biased by common coding conventions. Furthermore, natural language descriptions are often underspecified. The authors tackle two main questions:

  1. How do we accurately measure if a generated program matches a specification?
  2. Where in the agentic pipeline is information actually lost, and how can we recover it?

Methods

The researchers compared two distinct pipeline architectures:

  • Pipeline v1 (Original): A traditional "generate-then-check" approach. It parses requirements, generates Solidity code, runs an LLM auditor to find security flaws, and then uses a "refiner" agent to patch the code.
  • Pipeline v2 (Elicitation-first): This novel approach prioritizes front-loading information. Before any code is written, a "checklist elicitor" asks the user (or a simulated client) clarifying questions. It then generates code and checks it against an independently written Python-based "rulebook" (reference model) using a differential harness.
Figure 1: The two pipelines compared in this paper. (a) The original pipeline parses requirements without consulting their author, rewrites code in response to LLM security findings and grades itself with an LLM rubric. (b) The elicitation-first pipeline asks the client checklist questions before generation, checks the contract against an independently written reference model, shows each disagreement to the client as a blind choice and reports security findings without rewriting code. Badges give measured effects; percentages are transaction-level fidelity to the original contract on 500 held-out contracts (§4.5).

To ensure their evaluation was meaningful, they used Implementation Fidelity: replaying 60 random transaction sequences on both the original contract and the generated one. A step is only "correct" if the outcome (success/revert) and all resulting state changes (balances, variables) match exactly.

Datasets

The study utilized a corpus of 600 smart contracts (100 for development, 500 for held-out replication) sourced from the FSM-SCG dataset.

  • Descriptions: For each contract, an LLM generated a short "client-style" description (~140 words) and a "complete" requirements document (~1,700 words).
  • Simulated Client: GPT-5.4 was used to act as the "author," reading the original contract to answer the agent's clarifying questions.

Key Insights and Ideas

  • The "Conventions Shortcut": Standard benchmarks often reward models for knowing common patterns (e.g., "only the owner can withdraw") rather than actually understanding the specific prompt. The authors found that a model given only function names could catch 87.5% of conventional mistakes.
  • Auditing is a Value-Sinker: Surprisingly, the automated security-repair loop in Pipeline v1 actually decreased fidelity by 12.8 points. The "refiner" often broke intended behavior while trying to fix perceived (but sometimes non-existent) vulnerabilities.
  • Front-loading Information is King: Information gathered before generation matters significantly more than any multi-agent "reasoning" or "refinement" steps performed after the code is written.
  • Elicitation Efficiency: You don't need an endless Q&A session. Just 3 to 5 well-chosen questions (focused on roles, failure conditions, and "anything unusual") recovered 60-75% of the gain seen from a complete requirements document.
Figure 3: Question budget (92 contracts, transaction-level fidelity). (a) Asking the checklist in its fixed order gains little until question 6. Ordering questions by their measured value, learned on the other half of the contracts, reaches 68% with three questions and 71% with five. Points show adaptive agents at their mean number of questions. (b) Gain from each question asked alone; a few questions carry most of the value. Bands and whiskers are 95% intervals.

Key Results

  • Pipeline Performance: The elicitation-first pipeline reached 72.9% transaction-level fidelity on 500 held-out contracts. This is a massive leap over the original pipeline's 39.3% and even beats simple one-shot generation (53.9%).
  • Instrument Validity: The authors proved that common tools are unreliable for measuring intent. For example, a popular static analyzer (Slither) caught only 1.5% of "planted" logic mistakes (like removing a required check).
  • Reference Model Accuracy: Reference models written from complete descriptions caught 94.3% of conventional mistakes, but their performance dropped significantly when faced with real-world generation defects where the description was silent.

Limitations and Future Work

  • Simulated vs. Human Authors: The "client" in this study was an LLM with perfect knowledge of the original contract. Real humans might be inconsistent, forgetful, or provide ambiguous answers.
  • Originals as "Ground Truth": The study assumes the original contract represents the "intent." In reality, the original might itself contain bugs or missing access controls.
  • Perfect Pass Rate: Even the best pipeline only produced contracts that passed every sampled test in 20% of cases. There is still a significant "last mile" problem in reaching production-grade reliability.
  • Future Directions: The authors suggest moving toward studies with human commissioners and exploring "security as questions"—where a security agent asks the user for clarification rather than silently rewriting their code.

Summary updated

Abstract

AI agents can now write software from a plain-English description, but code that runs is not the same as code that does what was asked. This gap matters most for smart contracts, the programs behind Web3 applications that move money on a blockchain and cannot be fixed once deployed. We ask how to check whether an AI-written contract behaves as it should, and where the requested behavior gets lost. We design an evaluation built on execution rather than opinion. We replay the same transactions on an AI-written contract and on the original it should reproduce and measure how often they agree. We test every grading tool on planted mistakes, compare against an answer key written without the description, and audit our own scorer until known wrong answers cannot pass. Common methods, such as AI judges and standard analysis tools, miss obvious mistakes, and scoring choices matter too. Checking only the functions every system includes hid about half the difference between pipelines. We then propose an elicitation-first architecture. Before writing code, the agent asks the client a short checklist of questions, checks the contract against an independently written executable answer key, and returns disagreements to the client instead of silently rewriting code. Asking first matters far more than reviewing or patching afterwards, across three AI models, while automated security fixes often break requested behavior. A few well-chosen questions recover much of what a short description omits, and their value can be estimated without the original contract. To our knowledge, this is the first study in smart contract synthesis to gather missing requirements before generation and check the result against an independently written executable model. The lessons extend to any setting where AI agents turn requests into programs. We release our tools, 600 tasks and all results.
BibTeX
@misc{goel2026measuring,
  author = {Abhinav Goel and Agostino Capponi and Alfio Gliozzo and Chaitya Shah},
  title = {{Measuring Where Intent Gets Lost in Agentic Smart Contract Synthesis}},
  year = {2026},
  eprint = {2610.37830},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.37830},
}
Figure