Anatomy of an Agent Harness: Decisions, Invariants and Corner Cases

Resham JoshiarXiv 2026 · arXiv

TL;DR

While the industry focuses on the "brain" (the Large Language Model), this paper shines a light on the "harness"—the critical software wrapper that turns model outputs into real-world actions. By formalizing 30 key design decisions into testable invariants, the author provides a rigorous engineering framework to prevent the "corner case" failures that plague modern AI agents.

What Problem Does This Paper Address?

In the world of AI agents, a language model on its own is just a text generator. To actually do something—like delete a file, browse the web, or run a shell command—it needs a harness. This is the program that manages the "turn loop": it sends requests to the model, parses the replies, executes tools, handles network drops, and maintains the conversation state.

The problem is that harness design is currently more "vibes" than engineering. Most design decisions are buried in code and rarely documented. Consequently, every developer ends up rediscovering the same painful corner cases: What happens if the model's reply is cut off mid-tool-call? How do we handle a "Stop" command during a long-running process? This paper seeks to move agent development from ad-hoc scripting to a systematic discipline.

Methods

The author introduces a structured approach to harness design centered around 30 Decision Cards categorized into six functional groups: The Turn, Failures, The Prompt, State, Permission, and Display.

Each card follows a strict format:

  • The Decision: The specific choice the builder must make (e.g., "How a turn ends").
  • Options: The trade-offs between different implementation paths.
  • The Invariant: A universal property that should hold true regardless of the chosen option (e.g., "Every turn ends with exactly one recorded end reason").
  • Corner Cases: The specific, often weird, scenarios that break the invariant.
  • The Test: A concrete method to verify the invariant.

To support this, the paper defines a System Model with clear Trust Boundaries. It treats the model as an untrusted entity—its output is a request, not a decision. The author also introduces a testing methodology using "scripted providers" (mock LLMs that replay specific reply shapes) and fault injection to simulate edge cases like rate limits and malformed JSON deterministically.

Datasets

This is not a traditional machine learning paper, so it doesn't use standard training datasets like Common Crawl. Instead, it relies on:

  • Crewroom: The author's own desktop agent harness, used as the primary "worked example."
  • Reference Harnesses: Manual testing and observation of 10 other harnesses (both open-source and proprietary) to identify common patterns and failures.
  • The Checklist: A newly introduced benchmark of 30 items used to evaluate the reliability and "correctness" of a harness's logic.

Key Insights and Ideas

  • The "Two Readers" Problem: One of the most common failure modes occurs when two parts of a system interpret the same input differently. For example, if a security "checker" and the "shell" disagree on how to parse a command string, an agent might bypass a sandbox.
  • Invariants over Benchmarks: While most AI research focuses on accuracy (did the agent solve the task?), this paper focuses on correctness (did the harness behave predictably?). An agent might be "smart" but still crash because the harness didn't handle a "discarded attempt" correctly.
  • The "Behaviour Lock": The author advocates for recording every observable event in a turn (request bodies, log lines, exit codes) to create a "lock." When refactoring the harness, developers can ensure the system remains byte-identical in its behavior, catching regressions that traditional unit tests miss.
  • Trust Boundaries: The paper systematizes the "attack surface" of an agent. It identifies six types of input that cross into the trusted base, including model output, tool results, and even "other agents' text."

Key Results

The paper identifies 83 specific invariant clauses for agent harnesses.

  • Evidence Tags: Of these 83 clauses, 56 are "Tested" (verified in the author's harness), 12 are "Believed" (observed elsewhere or reasoned), and 15 remain "Open" (unresolved or known to be false).
  • Self-Assessment: When applying the 30-item checklist to the author's own harness (Crewroom):
  • 11 items held (e.g., tool call matching, reasoning isolation).
  • 14 items did not hold (e.g., handling timed-out commands that continue running in the background).
  • 5 items were not tested.
  • Failure Taxonomy: The paper maps corner cases to six "cause classes," providing a diagnostic manual for builders to find "sibling" bugs once one is discovered.

Limitations and Future Work

  • Single Rater: The evaluation was performed by the author alone; there was no second rater to provide an objective score.
  • Scope: The paper explicitly does not evaluate "answer quality," cost, or performance. It is purely about the structural integrity of the harness.
  • Platform Specificity: The findings are rooted in desktop/command-line harnesses; web-based or mobile-specific corner cases might differ.
  • Future Directions: The author suggests several "open questions," such as how to handle approval requests that survive a system restart and whether look-alike (homoglyph) characters in model output should be normalized for security.

For those building the next generation of AI assistants, this paper serves as a vital "pre-flight checklist" to ensure your agent doesn't just think well, but acts reliably.

Summary updated

Abstract

An agent harness is the program around a language model that turns its replies into actions: it builds each request, runs the tools the model asks for, decides what the model may do without asking, stores the conversation, recovers from failures, and shows a person what happened. Most of these decisions are made in code and rarely written down, so each harness meets the same corner cases again. This paper writes them down as invariants to test. It gives a system model with trust boundaries; 30 decision cards in six groups, each with options and costs, an invariant, the corner cases that break it, a test and an evidence tag; six cause classes for failures; a test method built on scripted providers, recorded behaviour and planted changes; and a checklist. The cards come from building one desktop harness, Crewroom, the author's own, and from manually testing ten other harnesses and studying open-source harnesses as reference; some patterns in the cards were adopted from those, so the catalogue is not independent of them. Of the 83 invariant clauses, 56 are tagged Tested, 12 Believed and 15 Open. The checklist is applied to the author's harness as one worked example, by its builder with no second rater: of its 30 items, 11 hold, 14 do not and 5 were not tested. It is not a ranking. The paper makes no claims about answer quality or cost or other platforms, and does not evaluate or rank other harnesses.
BibTeX
@misc{joshi2026anatomy,
  author = {Resham Joshi},
  title = {{Anatomy of an Agent Harness: Decisions, Invariants and Corner Cases}},
  year = {2026},
  eprint = {2611.19960},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.19960},
}
Figure