Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them

Shaohuang Wang, Jinghao Wen, Xulei Zhang, Xun JiaoarXiv 2026 · arXiv

TL;DR

This paper moves beyond simple "failure mode" labels for LLM agents by introducing a trajectory-centered framework to trace how errors originate, propagate, and persist. The study reveals that faults enter early (median 18% into a run), and in 72% of failures, the evidence needed to catch the error was already present in the logs but ignored by the agent.

What Problem Does This Paper Address?

As LLM agents tackle increasingly complex, "long-horizon" tasks—like managing a simulated business or navigating multi-step software engineering problems—traditional failure analysis is falling short. Currently, when an agent fails, we usually just slap on a label like "task derailment" or point to the specific step where the crash happened.

The authors argue this is like diagnosing a plane crash by only looking at the crater. It doesn't tell us:

  1. Where the error actually entered: The "decisive step" identified by humans is often just the symptom of a wrong turn taken much earlier.
  2. How the error traveled: How a single false belief in a "fact sheet" contaminates the rest of the trajectory.
  3. Why it wasn't caught: Was the information missing, or did the agent simply ignore contradicting evidence?

Methods

The researchers developed a Trajectory-Centered Dependability Framework to perform a deep forensic analysis of agent failures.

Fig. 1. The trajectory-centered dependability framework. P0–P7 locate operations in execution; R1–R6 lo- cate the state or resource affected; H1–H7 describe possible dynamics across these coordinates. Recovery is conditional and audit can span the task. The lower panel is an analytical synthesis: controls can interrupt the paths, and no single dynamic implies failure.

Key components include:

  • The Episode Record: A structured analytical record that links a candidate fault to its origin, propagation, detection, and eventual consequence.
  • Two-Coordinate Mapping: Events are located by Operational Phase (e.g., Deliberation, Execution, Verification) and State Domain (e.g., Task State, Coordination State, Environment).
  • Annotation Protocol: A 38-field protocol applied to 95 failed runs from multi-agent systems. The annotation was performed by an independent LLM (Claude) following strict rules to ensure consistency.

Datasets

The study analyzed failed trajectories from two prominent public corpora:

  • Who&When: 184 failed runs involving systems like Magentic-One and CaptainAgent teams.
  • MAD (Multi-Agent Diagnostics): 19 traces from various frameworks including MetaGPT and ChatDev.
  • The final analyzed sample consisted of 95 failed runs after filtering for length and procedural validity.

Key Insights and Ideas

  • Errors are "Early Birds": Faults enter very early in the trajectory (median at 0.18 of the run length), while the visible symptom doesn't appear until much later (median gap of 0.67 of the run).
  • The Evidence is Hiding in Plain Sight: In 72% of failed runs, the trajectory already contained an observation or message that contradicted the erroneous belief before the failure occurred. In 88% of those cases, no agent acted on that evidence.
  • State Contamination is the Real Killer: Most faults (71 out of 95) corrupt the "Task and Knowledge State" (R2). Once a false belief is written into a plan or fact sheet, it acts as a "poison pill" that persists even if the agent tries to "re-plan."
  • The "Harness Tax": A surprising 45% of hand-crafted failures in the Who&When corpus were actually caused by provider content-filter crashes (HTTP 400 errors) rather than agent logic, yet these were often attributed to the agents themselves in the original dataset.

Key Results

The authors categorized the "Kind" of origin events to see what actually goes wrong at the start of a failure chain:

Table 3. Kind of the origin event.
  • Decision Faults (42%): The agent had adequate information but made the wrong choice.
  • Fabrications (26%): The agent asserted a fact or tool result it never actually observed.
  • Ineffective Recovery: While 31% of runs detected a deviation, zero of the resulting responses successfully removed the deviated state. Typical "re-planning" changes the next action but keeps the toxic belief that caused the loop in the first place.
  • Control Reach: The study found that "Independent Judges" (who read the evidence the worker ignored) and "Environment Re-verification" would have reached the most episodes (74% and 43% respectively).

Limitations and Future Work

  • LLM-as-Annotator: While the authors used a rigorous protocol and mechanical checks, they acknowledge the lack of human inter-annotator agreement in this version.
  • Sandbox Constraints: The studied corpora were mostly question-answering or sandboxed programming. This meant "External Impact" (errors that change a real-world system in an undoable way) couldn't be fully explored.
  • Counterfactuals: The study identifies which controls would have reached an error, but it doesn't re-run the agents with those controls in place to measure actual success rate improvements.
  • Future Direction: The authors call for "Dependency-Guided Attribution" and better "Semantic Memory" controls that track the provenance and uncertainty of every stored fact.

Summary updated

Abstract

When an LLM agent fails part-way through a long task, the usual analysis records a symptom: a failure-mode label, or the agent and step judged responsible. It does not say where the error entered, which state carried it to the symptom, whether anything in the trajectory contradicted it, or which control could have caught it. We define an episode record that ties a candidate fault to its origin, propagation, detection, response and consequence, located by operational phase and state domain, and that names the control families whose observation and reach cover it. We turned the record into a written annotation protocol and applied it to 95 failed runs of six multi-agent systems from the public Who&When and MAD corpora, reading each trajectory in full. Faults enter early (median at 0.18 of the run), mostly as wrong decisions or fabrications over task state, and in 37 of 88 Who&When runs before the step the corpus' human labels mark as decisive. In 72% of failed runs the trajectory already contained evidence against the error before the failure; in 88% of those no agent acted on it. Something flagged a deviation in 31% of runs, but no response removed the deviated state. Independent judges reading the unused evidence, task contracts fixed at admission, and re-checking beliefs against the environment reach most episodes; restoring the agent's own state would have sufficed in 35 of 95 runs. A text scan of the raw corpus also shows that 45% of the hand-crafted Who&When failures ended with a provider content-filter crash after which the harness emitted a final answer no agent produced, a failure class that neither symptom taxonomies nor responsible-agent labels represent. Protocol, records and scripts are included.
BibTeX
@misc{wang2026dependable,
  author = {Shaohuang Wang and Jinghao Wen and Xulei Zhang and Xun Jiao},
  title = {{Dependable Long-Horizon LLM Agents: A Trajectory-Centered Empirical Study of Where Errors Enter and Which Controls Reach Them}},
  year = {2026},
  eprint = {2611.13254},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.13254},
}
Figure