RECEIPT: Measuring and Repairing the Return Assumption of LLM Agents in Web Service Operations

Boyang Yang, Ziyao Yang, Yu Zhang, Duo Chai, Xiaoming Chen, Haoye TianarXiv 2026 · arXiv

TL;DR

This paper identifies and repairs the "return assumption"—the flawed belief that an LLM agent's tool call always finishes its requested action if it returns a success code. By introducing RECEIPT, a system that monitors process states via system calls and a model-checked protocol, the authors reduce false success claims by 73.5% and cut token costs by nearly a quarter.

What Problem Does This Paper Address?

When LLM agents operate web infrastructure via the command line, they rely on a "return assumption": that every tool call returns within a reasonable time and has actually completed its task. However, many command-line tools (like rm -i or git rebase) pause to ask for confirmation or credentials.

Current agent "harnesses" (the environments where agents run) handle these prompts poorly. They either:

  1. Close the input: The program sees an "End of File" (EOF), takes a default action (often doing nothing), and returns a success status (0). The agent thinks it succeeded when it actually failed—a "false success."
  2. Keep the input open: The program hangs indefinitely waiting for an answer. The agent eventually times out, wasting time and expensive tokens.

This creates a massive gap in reliability and distorts benchmarks. If a leaderboard ranks LLMs based on a harness that closes input, it might be rewarding models that are simply "lucky" with defaults rather than those that are truly capable.

Figure 1: Comparison of how different input handling strategies affect a database migration task.

Methods

The authors propose RECEIPT, a three-layered approach to "repairing" the return assumption:

  • Layer 1: Wait Detection via System Calls. Instead of just looking at text output (which might be buffered or missing), RECEIPT samples kernel data every 0.1 seconds. It checks if a thread is blocked on a specific file descriptor (like a terminal or pipe) and confirms a "wait" state if the process uses minimal CPU and produces no output for one second.
  • Layer 2: Model-Checked Intervention. Managing a persistent terminal is tricky because of race conditions—an agent might type a command just as a prompt appears. RECEIPT uses a formal automaton to ensure that input is only delivered when the agent has "seen" the current state, preventing "stale" inputs from being misread by the program.
  • Layer 3: Formal Verification. The authors used TLA+ and a model checker to verify that their protocol correctly orders input, interruptions, and reports, ensuring no "forbidden" states (like interrupting a process that just exited) are reachable.

Datasets

The study introduces several new resources to the community:

  • LITMUS: A catalog of 158 command-line invocations (e.g., docker system prune, ssh-keygen) used to test how programs behave when no one answers their prompts.
  • WaitAtlas: A dataset of 4,855 commands from 668 package families, used to see if "waits" can be predicted before a command even runs (spoiler: it's hard).
  • TRIPWIRE: A new benchmark consisting of 25 task pairs. Each pair has a "control" version (no prompt) and a "trap" version (identical task but triggers a prompt). This allows researchers to isolate exactly how much a prompt costs an agent.

Key Insights and Ideas

  • The Harness is Part of the Model: An agent's performance is inextricably linked to its execution environment. The authors found that changing the input connection moved Llama-3.3-70B from 7th place to 3rd place among eight models.
  • Closed Input is Not "Safe": Many developers assume closing input prevents accidental destructive actions. In reality, 11.5% of tested commands in LITMUS returned a success code while silently skipping the requested action.
  • The "Silent Tax" of Terminals: Running an agent in a plain terminal without wait detection multiplies token usage by 2.9x because the agent repeatedly polls or waits for timeouts on stalled processes.
Figure 5: LLM rankings shift significantly depending on whether the harness uses closed input or a plain terminal.

Key Results

  • Reliability Boost: Across five different LLMs, RECEIPT reduced false claims of success by 73.5% compared to standard closed-input harnesses.
  • Efficiency: Compared to a plain terminal, RECEIPT used 24.4% fewer tokens by identifying waits early and preventing unnecessary retries.
  • High Precision: RECEIPT correctly reported 85.0% of waits in the LITMUS dataset with zero false positives.
  • Low Overhead: In a commercial agent runtime, RECEIPT's checks added a median delay of only 0.54 milliseconds per shell call, making it practical for production use.
Figure 2: Outcomes of command-line invocations under different input conditions, showing how terminals lead to "Hangs" while closed input leads to "Fails" or "Skips."

Limitations and Future Work

The authors acknowledge that while RECEIPT is highly effective for CLI-based agents, it currently focuses on local process trees. Future work could extend these techniques to:

  • Remote Execution: Detecting waits over SSH or in distributed systems where kernel sampling is more difficult.
  • Complex TTY interactions: Handling more sophisticated full-screen applications (like vim or top) that use complex terminal escape sequences.
  • Benchmark Standardization: The authors strongly urge benchmark maintainers to report and fix input connection settings to ensure LLM rankings reflect actual reasoning capability rather than harness quirks.

Summary updated

Abstract

LLM agents operate Web infrastructure from the command line and rely on a return assumption, namely that each tool call returns within a bounded time and with its requested action finished. Many commands, however, ask a question before they act, and the agent's harness either closes the program's input or keeps it open as a terminal at which nobody types. With closed input, the program can return status 0 without the requested action. With open input, it waits until a time limit, and the agent pays in time and tokens. To the best of our knowledge, this paper is the first to measure and repair the return assumption. We collect 91,469 production tool calls and 89,047 public agent trajectories, test how 8 widely used harnesses launch commands, and build Litmus, a catalog of 158 checked command-line invocations, and Tripwire, a benchmark of 25 task pairs that differ only in whether a question guards the requested action. We then propose RECEIPT, which keeps the input open, reads what each process waits for from system calls, and returns this state to an unmodified agent through a model-checked protocol. On Litmus, closed input causes 11.5% of tested invocations to return status 0 without performing the requested action, and 7 of 8 harnesses close input when no terminal is attached. An unanswered question at a plain terminal multiplies a task's token use by 2.9, and the input connection moves an LLM from 7th to 3rd place among 8. Across 5 LLMs, RECEIPT reduces false claims of success by 73.5% and completes all 25 guarded tasks on a second harness. An agent's reliability therefore depends on what its harness lets it observe as well as on its LLM. Agent builders should report what each tool call did, not only how it ended, and benchmark maintainers should fix and report the input connection before they rank LLMs.
BibTeX
@misc{yang2026receipt,
  author = {Boyang Yang and Ziyao Yang and Yu Zhang and Duo Chai and Xiaoming Chen and Haoye Tian},
  title = {{RECEIPT: Measuring and Repairing the Return Assumption of LLM Agents in Web Service Operations}},
  year = {2026},
  eprint = {2611.01930},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.01930},
}
Figure