When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems

Shihao Weng, Longtian Wang, Yang Feng, Mingfei Cheng, Jia Liu, Chao Shen, He Ye, Federica Sarro, Xiaofei XiearXiv 2026 · arXiv

This empirical study of 40 open-source agentic systems reveals a critical "adequacy gap" in how developers handle Large Language Model (LLM) outputs. While most systems include some checks, 63% of necessary safety and functional constraints are unmet, leaving agents vulnerable to exploits that can be stopped with simple code-level enforcement.

What Problem Does This Paper Address?

In traditional software, quality is built on a foundation of specifications: pre-conditions, post-conditions, and types that define exactly what a unit of code must do. Agentic systems—programs that use LLMs to make decisions, write code, or use tools—break this foundation. Because LLM behavior is non-deterministic and steerable by input (e.g., prompt injections), the "logic" of the program is no longer fixed.

The authors argue that every LLM call site needs a contract: a set of guarantees enforced by the surrounding code to ensure the model's output is safe and correct. Currently, there is little understanding of how developers specify these contracts, whether they are adequate, or what happens when they fail. This paper provides the first large-scale empirical look at these "call-site contracts."

Methods

The researchers developed a novel three-stage pipeline to analyze 496 LLM call sites (see Figure 2).

![Fig.

  1. Overview of the study.](/figures/art_a5d3096875c54c45855cdec2daafd254)
  2. Call-Site Analysis: They used static analysis to locate LLM calls and traced how the model's output flows through the program. They categorized outputs based on their operation (e.g., generating code, selecting a tool) and how they are consumed (e.g., driving control flow).
  3. Blind Contract Reconstruction: To measure "adequacy" without bias, they used a multi-agent system (Figure 3). One agent derived an expected contract based only on a functional specification of what the call should do. A separate agent extracted the implemented contract from the actual source code. Comparing these two reveals "contract gaps."
  4. Dynamic Validation: They manually tested whether these gaps lead to real-world vulnerabilities by creating Proof-of-Concept (PoC) exploits and verified if code-level fixes could mitigate them.

![Fig.

  1. Multi-agent contract reconstruction for one responsibility unit.](/figures/art_6efe5fb2cb6d4b6c9b060f146ce5ebd2)

Datasets

The study analyzed 40 popular open-source agentic systems (Table 1) totaling 13.4 million lines of code across four languages (Python, TypeScript, Rust, Go). These systems span domains like software engineering (e.g., OpenHands, Aider), personal assistants (e.g., Khoj), and domain-specific agents (e.g., FinRobot).

Table 1. Subject systems, grouped by task domain and sorted by GitHub stars.

Key Insights and Ideas

  • The "Adequacy Gap": The most striking finding is that the mere presence of checks is misleading. While 82% of call sites have some code-level checks, only 5.3% satisfy their entire expected contract. Most checks verify the form of a reply (e.g., "is it valid JSON?") but ignore its meaning (e.g., "is this tool call allowed?").
  • Delegating Critical Logic: LLMs aren't just for chat; 48% of their outputs reach control flow or direct execution. Alarmingly, 73% of these critical outputs never reach a human for review before taking effect.
  • Prompt-Only "Enforcement": Developers often rely on the prompt to enforce constraints (e.g., "only use the following tools"). However, 67% of these prompt-phase constraints are never verified in code. If the model hallucinations or is subverted, the program has no fallback.
  • Recurring Failure Patterns: The authors identified ten recurring patterns that explain 99% of contract gaps. A major one is "fail-open handling," where a system defaults to "allow" or "continue" if a safety check fails or returns an unexpected format.

Key Results

  • 63% of expected constraints are unmet across the studied systems.
  • Post-conditions are the weakest link, with a 72% gap rate. Programs check replies often, but rarely cover everything the reply needs to guarantee.
  • Exploitability: Out of 1,697 identified gaps, 290 were found to be technically exploitable. The researchers successfully triggered 38 distinct exploit mechanisms at runtime.
  • Simple Fixes Work: In all 38 triggered cases, a minimal code fix (often just a few lines) that enforced the missing expected constraint was enough to stop the exploit completely.

Limitations and Future Work

The authors acknowledge that "expected contracts" are normative reconstructions; different analysts might propose slightly different requirements. Additionally, their static analysis might miss call sites reached through complex dynamic dispatch or reflection.

Future directions suggested by the paper include:

  • Framework-level defaults: Moving enforcement from individual apps to frameworks (e.g., tool dispatchers that automatically reject unoffered tools).
  • Consumer-aware detectors: Building security tools that flag dangerous patterns, like "fail-open" safety gates.
  • Contracts as test oracles: Using derived expected contracts to automatically generate test cases and runtime monitors for agents.

Summary updated

Abstract

Software quality has long rested on specifications that state what code must do, and on testing and verification that check that the code does it. Agentic systems change this foundation: they replace parts of their logic, such as choosing the next tool, judging whether an action is allowed, or writing code to run, with calls to a large language model (LLM). The program still relies on what this logic must guarantee, but the model's behavior is not fixed by code, varies across runs, and can be steered by its input. The guarantee must therefore come from the program around each LLM call site, as a contract of pre-conditions, prompt constraints, and post-conditions. We present an empirical study of these contracts in 496 LLM call sites of 40 popular open-source agentic systems (13.4 MLOC, four languages). Using a multi-agent pipeline validated by 17,428 human judgments, we trace where every model output flows, derive an expected contract for each call site without access to its code, and compare it with the contract that the code implements. We find that 48% of LLM outputs reach control flow or execution, and 73% of these never reach a person. Programs enforce 78% of their constraints in code, but in the prompt, 67% of the constraints are only requested from the model. Although 82% of call sites are checked before and after the call, 63% of the expected constraints are unmet and only 5.3% of call sites satisfy their whole contract; ten recurring patterns explain 99% of the gaps. 38 mechanisms behind these gaps trigger at runtime, and a small code fix that enforces the expected constraint stops all 38. Our results show that the presence of checks says little about their adequacy, and we derive implications for agent developers, framework maintainers, and researchers.
BibTeX
@misc{weng2026when,
  author = {Shihao Weng and Longtian Wang and Yang Feng and Mingfei Cheng and Jia Liu and Chao Shen and He Ye and Federica Sarro and Xiaofei Xie},
  title = {{When Code Becomes an LLM Call: An Empirical Study of Call-Site Contracts in Agentic Systems}},
  year = {2026},
  eprint = {2610.24545},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.24545},
}
Figure