TL;DR
This paper demonstrates that for agentic code generation, gathering missing requirements before writing code is far more effective than trying to audit or repair code after the fact. By introducing a rigorous evaluation framework based on exact transaction replay and "elicitation-first" pipelines, the authors show that a few well-chosen questions can boost smart contract fidelity from 39.3% to 72.9%.
What Problem Does This Paper Address?
Large Language Model (LLM) agents are increasingly used to synthesize code from natural language. However, "code that compiles" is a low bar; the real challenge is "code that does what the user intended." This is especially critical for smart contracts, which manage high-value assets on blockchains and cannot be patched once deployed.
Current evaluation methods often rely on LLM-as-a-judge or static analysis, but these are "unvalidated instruments"—they often fail to catch subtle logic errors and can be biased by common coding conventions. Furthermore, natural language descriptions are often underspecified. The authors tackle two main questions:
- How do we accurately measure if a generated program matches a specification?
- Where in the agentic pipeline is information actually lost, and how can we recover it?
Methods
The researchers compared two distinct pipeline architectures:
- Pipeline v1 (Original): A traditional "generate-then-check" approach. It parses requirements, generates Solidity code, runs an LLM auditor to find security flaws, and then uses a "refiner" agent to patch the code.
- Pipeline v2 (Elicitation-first): This novel approach prioritizes front-loading information. Before any code is written, a "checklist elicitor" asks the user (or a simulated client) clarifying questions. It then generates code and checks it against an independently written Python-based "rulebook" (reference model) using a differential harness.
To ensure their evaluation was meaningful, they used Implementation Fidelity: replaying 60 random transaction sequences on both the original contract and the generated one. A step is only "correct" if the outcome (success/revert) and all resulting state changes (balances, variables) match exactly.
Datasets
The study utilized a corpus of 600 smart contracts (100 for development, 500 for held-out replication) sourced from the FSM-SCG dataset.
- Descriptions: For each contract, an LLM generated a short "client-style" description (~140 words) and a "complete" requirements document (~1,700 words).
- Simulated Client: GPT-5.4 was used to act as the "author," reading the original contract to answer the agent's clarifying questions.
Key Insights and Ideas
- The "Conventions Shortcut": Standard benchmarks often reward models for knowing common patterns (e.g., "only the owner can withdraw") rather than actually understanding the specific prompt. The authors found that a model given only function names could catch 87.5% of conventional mistakes.
- Auditing is a Value-Sinker: Surprisingly, the automated security-repair loop in Pipeline v1 actually decreased fidelity by 12.8 points. The "refiner" often broke intended behavior while trying to fix perceived (but sometimes non-existent) vulnerabilities.
- Front-loading Information is King: Information gathered before generation matters significantly more than any multi-agent "reasoning" or "refinement" steps performed after the code is written.
- Elicitation Efficiency: You don't need an endless Q&A session. Just 3 to 5 well-chosen questions (focused on roles, failure conditions, and "anything unusual") recovered 60-75% of the gain seen from a complete requirements document.
Key Results
- Pipeline Performance: The elicitation-first pipeline reached 72.9% transaction-level fidelity on 500 held-out contracts. This is a massive leap over the original pipeline's 39.3% and even beats simple one-shot generation (53.9%).
- Instrument Validity: The authors proved that common tools are unreliable for measuring intent. For example, a popular static analyzer (Slither) caught only 1.5% of "planted" logic mistakes (like removing a required check).
- Reference Model Accuracy: Reference models written from complete descriptions caught 94.3% of conventional mistakes, but their performance dropped significantly when faced with real-world generation defects where the description was silent.
Limitations and Future Work
- Simulated vs. Human Authors: The "client" in this study was an LLM with perfect knowledge of the original contract. Real humans might be inconsistent, forgetful, or provide ambiguous answers.
- Originals as "Ground Truth": The study assumes the original contract represents the "intent." In reality, the original might itself contain bugs or missing access controls.
- Perfect Pass Rate: Even the best pipeline only produced contracts that passed every sampled test in 20% of cases. There is still a significant "last mile" problem in reaching production-grade reliability.
- Future Directions: The authors suggest moving toward studies with human commissioners and exploring "security as questions"—where a security agent asks the user for clarification rather than silently rewriting their code.
Summary updated