Beyond Prompts: Runtime Contracts for Coding Agents

Reda HeddadarXiv 2026 · arXiv

TL;DR

This paper argues that "asking nicely" via prompts is insufficient for steering coding agents in industrial or safety-critical environments. By implementing runtime executable contracts—hard gates that block disallowed actions before they execute—researchers achieved zero rule violations across 104 test runs, significantly outperforming prompt-based guidance without sacrificing the agent's ability to solve tasks.

What Problem Does This Paper Address?

As LLM-based coding agents move from suggesting code to actively managing repositories (running shell commands, editing files, and submitting PRs), they gain the power to cause real-world side effects. In regulated industries (e.g., automotive, aerospace, or finance), software development must follow strict engineering and security rules (like "don't use eval()" or "don't modify test files").

Currently, we steer these agents using prose prompts. However, prompts offer two major failures:

  1. Lack of Guarantee: A model might simply ignore a prompt.
  2. Lack of Evidence: Prompts provide no verifiable audit trail that rules were actually followed.

The author investigates whether moving from "soft" prompts to "hard" executable contracts can provide the enforcement and evidence required for professional engineering standards.

Methods

The researcher developed a framework using executable contracts: machine-readable rules (defined in YAML) that specify triggers, checks, and failure policies.

The study compared four primary conditions:

  • C0 (Control): No rules provided.
  • C1 (Prompt): Rules provided as text within the agent's prompt.
  • C3 (Post-Generation Check): The agent completes its task, then rules are checked. If they fail, the agent gets feedback and tries to revise.
  • C4 (Runtime Enforcement): A "gate" sits between the agent and the environment. Every shell command or file edit is intercepted and checked before it runs. Disallowed actions are blocked immediately.

To evaluate success, the author used an independent oracle. Crucially, the oracle used different detection methods (e.g., Abstract Syntax Tree (AST) scans and import graph analysis) than the in-loop enforcement (which often relied on regex) to ensure the evaluation wasn't just "grading its own homework."

Datasets

To avoid the "data leakage" common in public benchmarks (where LLMs might have seen the answers during training), the author created a custom Python fixture repository.

  • Tasks: Freshly written feature requests involving a small web application (handlers, services, and a SQLite store).
  • Studies:
    • Study 1 (Exploratory): 8 tasks, 240 runs using Gemini 3.1 Flash-Lite.
  • Study 2 (Confirmatory): 14 tasks, 112 runs using DeepSeek v4.1 Flash.
  • Study 3 (Targeted): 18 tasks specifically designed to trigger known failure modes (like recursive deletes), 108 runs using DeepSeek v4.1 Flash.

Key Insights and Ideas

  • Detection is not Prevention: The biggest "aha" moment is the distinction between C3 and C4. If an agent runs rm -rf /, a post-generation check (C3) can detect the error, but it cannot "un-run" the command. Runtime enforcement (C4) is the only way to handle irreversible actions.
  • Prompting is Uneven: Prompts worked well for some rules (like "don't touch the test file") but failed miserably for others. For example, when a prompt said "don't use recursive delete" but the system prompt said "you can use /tmp for scratch files," the agent often got confused and deleted its scratch directory recursively anyway.
  • The "Functional Tax" is Low: A common fear is that strict rules will "break" the AI's ability to think or solve problems. The data showed that even when the agent was blocked and had to try a different approach, its success rate (functional pass) did not drop significantly.
  • Oracles Need Humans: The study found a "blind spot" where both the enforcement regex and the automated oracle missed a specific type of SQL injection. This highlights that while automated gates are great, they aren't a total replacement for human review.

Key Results

  • Zero Violations: In the C4 (Runtime Enforcement) condition, the independent oracle detected zero violations across all confirmatory runs.
  • Prompt Failure: In Study 3 (targeted tasks), agents without rules violated policies in 31% of runs. Adding prompts (C1) reduced this, but still left a significant gap that only runtime enforcement closed.
  • Irreversible Actions: In Study 1, post-generation checks (C3) reduced file edit violations from 12 runs to 1, but left forbidden shell commands unchanged because they had already executed.
  • Cost: The overhead for running these checks was negligible, adding only about $0.004 USD per run in model fees.

Limitations and Future Work

  • Model Scale: The study used "Flash" and "Lite" models. While these are realistic for high-volume production, it remains to be seen if larger models (like GPT-4o or Claude 3.5 Sonnet) follow prompts well enough to make runtime enforcement less critical.
  • False Positives: The study focused on whether the gate blocked bad actions. Future work needs to measure "false blocks"—how often a gate stops a good action—which could frustrate developers.
  • Complexity: The rules tested were relatively simple (regex/AST-based). Enforcing more abstract architectural rules remains a challenge.
  • Scaling: The tests were performed on a small fixture. Testing on a massive, multi-million line repository would test the performance latency of the runtime checks.

Summary updated

Abstract

Coding agents act on repositories through shell commands and file edits, and teams steer them with written engineering rules. Industrial settings expect such rules to be enforced and their application to be evidenced, and a rule given as prompt text provides neither. We test whether executable contracts enforced while the agent works reduce rule violations compared with the same rules in the prompt, retrieved as guidance, or checked only after generation. Violations are scored by an oracle whose methods differ from the in-loop checks, and the oracle is validated by two human reviewers and a model reviewer. Across three studies on two model families (240, 112 and 108 runs), enforcement during execution produced no oracle-detected violation in 104 paired runs, against 21 to 55% of runs without rules, and no loss of functional correctness in the two confirmatory studies. In the confirmatory study on tasks aimed at known failure situations (Study 3, protocol hashed before the first run), enforcement lowered the share of violating runs by 31 percentage points relative to no rules (Holm-adjusted p = 0.014); the 17-point difference from prompted rules was not significant (p = 0.12), and neither was any contrast in Study 2, where only 3 of 14 tasks produced a violation. Prompted rules removed some violation types but not recursive deletes of scratch directories, and checks after generation repaired file edits but not commands that had already run. That enforcement satisfies the rules it enforces is expected by construction; what the data add is where prompting falls short, the absence of a functional cost, and the limits of the rules and of the oracle themselves. The studies use low-cost models and one small fixture.
BibTeX
@misc{heddad2026beyond,
  author = {Reda Heddad},
  title = {{Beyond Prompts: Runtime Contracts for Coding Agents}},
  year = {2026},
  eprint = {2611.19549},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19549},
}
Figure