This paper challenges the common assumption that an LLM agent’s performance is primarily limited by its context window size. Instead, the authors argue that the "harness"—the software wrapper that manages tool outputs and conversation history—is a leaky pipeline that silently destroys critical information before the model ever sees it.
TL;DR
LLM agents are often crippled not by small context windows, but by "harnesses" that naively slice tool outputs, leading to massive information attrition. This research shows that even when context windows are 95% empty, these engineering choices can double an agent's workload and wipe out entire classes of evidence.
What Problem Does This Paper Address?
In the world of LLM agents, we usually track two metrics: context window size and final accuracy. This paper identifies a critical "missing middle": information attrition.
When an agent uses a tool (like searching a database or reading a log), the output doesn't go straight to the LLM. It passes through a harness that serializes it into text and often "cuts" it to fit a budget. The authors found that many harnesses use "prefix slicing"—simply cutting the text at a fixed character limit. Because data formats like JSON have a predictable key order, this isn't a random thinning of data; it’s a deterministic deletion of everything at the end of a list. If your "smoking gun" evidence happens to be the last item in a JSON array, the model will never see it, even if the context window is nearly empty.
Methods
The authors propose a five-stage pipeline model to track how data survives (or dies) on its way to the model:
- S1: Produce → Deliver: Tool output is serialized and budgeted.
- S2: Deliver → Act: The agent operates within a turn/step budget.
- S3: Act → Retain: Conversation history is pruned.
- S4: Retain → Transfer: Findings are passed between worker agents.
- S5: Transfer → Persist: Data is (or isn't) saved across independent runs.
They instrumented a real-world agent designed to find root causes in Linux kernel traces. They then conducted a controlled experiment by re-introducing specific harness defects (like prefix slicing) one by one to isolate their impact. Finally, they surveyed seven popular agent frameworks (including LangChain and smolagents) to see if these "unsafe" cutting patterns are widespread.
Datasets
The study used:
- Linux Kernel Traces: Used for a fault-diagnosis task where an agent must identify a "culprit" process.
- Fault Classes: "Spawned" faults (processes starting during the trace) and "Resident" faults (processes already running).
- Production-Scale Replay: A real 21-container deployment trace used to validate findings at scale.
- Framework Survey: An audit of the source code for seven major LLM agent frameworks.
Key Insights and Ideas
- The "Delivery Cliff": Naive slicing creates a binary failure mode. As shown in Figure 1, once a tool result exceeds a character cap, the "late" fields in a JSON payload don't just get shorter—they vanish entirely, and the JSON structure itself breaks.
- The Compensation Tax: Interestingly, data loss doesn't always lead to a "wrong" answer. If the agent has multiple ways to find the truth, it will simply work harder. It makes more tool calls and consumes more tokens to compensate for the missing data.
- Empty Windows, Full Cuts: In 720 runs, the largest prompt used only 5.75% of the 400k context window. Despite this massive "headroom," the harness was still aggressively cutting tool outputs, destroying 34% of all results.
- Position is Destiny: Because JSON keys are often sorted, prefix slicing creates a systemic bias. In the authors' tests, one data field reached the model 99.3% of the time, while a neighboring field in the same payload reached it only 4.7% of the time.
Key Results
- Workload Doubled: Under the "S1" defect (prefix slicing), the agent required 43% more tool calls and roughly twice the computation to reach the same answer.
- Yield Collapse: The authors' "yield" metric (findings recorded divided by computations run) fell from 0.61 to 0.31 when the harness defect was present.
- Widespread Vulnerability: The survey found that two major frameworks (LangChain-core and smolagents) use position-based slicing by default. Most others don't cut at all, which simply pushes the problem further down the pipeline into the context window itself.
Limitations and Future Work
The authors acknowledge that their study is limited to one specific domain (kernel traces) and one model. While they proved that harness defects increase the "cost" of an answer, the link between these defects and final accuracy in the field remains correlational.
Future plans include: 1. Causal Accuracy Experiments: Testing "quiet" culprits that have no alternative discovery route to see if the "compensation tax" eventually turns into a hard failure. 2. Standardized Loss Reporting: Developing tools so that any agent harness can report its "information loss" alongside its accuracy scores. 3. Structure-Aware Cutting: Promoting "row-aware" or "structure-aware" slicing as a safer default for the industry.
Summary updated