Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching

Zerui Yang, Miaomiao Dong, Quan Xue, Chaorui Zhang, Xuanyu Zheng, Xiyuan Wang, Haolin Ruan, Xueyan Niu, Guangcheng Song, Weixi Zhang, Wei Han, Bo BaiarXiv 2026 · arXiv

TL;DR

Context management for LLM coding agents isn't just about token counts—it's an economic trade-off dictated by prefix caching. This study of over 20,000 runs reveals that common strategies like adding retrieved code or masking old history can actually increase total inference costs by 30–100% when cache discounts are factored in, often without improving task resolution.

What Problem Does This Paper Address?

LLM-based coding agents operate in a loop: they search for code, edit files, and run tests, appending each action to a growing conversation history. As the history grows, so does the cost of every subsequent request.

Researchers typically try to manage this by either adding context (retrieving relevant code to guide the agent) or removing context (masking or summarizing old tool outputs to shorten the prompt). However, modern LLM providers use prefix caching, where identical prompt prefixes are served at a significant discount (often 90% off).

The authors argue that current evaluations are flawed because they ignore these pricing dynamics. An "optimization" that reduces total tokens might actually break the cache, forcing the model to re-process the entire prompt at the expensive "uncached" rate.

Methods

The authors treat context management as a measurement problem, accounting for three distinct token streams:

  1. Uncached Input ($U$): Fresh tokens that must be processed.
  2. Cached Input ($C$): Tokens reused from a previous request's prefix.
  3. Output ($O$): Tokens generated by the model.

They define a cost function: $cost(c, p) = U + cC + pO$, where $c$ is the relative price of cached tokens and $p$ is the relative price of output tokens.

The study involved over 20,000 runs using open models (Qwen, Gemma, GLM) across various "scaffolds" (the environment the agent lives in). They tested two primary interventions:

  • Evidence Packages: Adding retrieved code snippets to the prompt.
  • Observation Masking: Hiding old tool outputs using different policies (rolling, batched, or a novel "break-even" rule).

Datasets

The researchers evaluated their agents on three major benchmarks:

  • SWE-bench & SWE-bench Verified: Real-world GitHub issues requiring repository-level edits.
  • Loc-Bench: A benchmark specifically for code localization (finding where the bug is).
  • Executable Sandboxes: Custom environments for Django and SymPy to test if agents can actually resolve bugs.

Key Insights and Ideas

  • The Cache Invalidation Penalty: Deleting an old output early in the conversation history might save tokens, but it "breaks" the cache for everything that follows. If the following suffix is large, you end up paying for it all over again at the full uncached price.

    Figure 2: Deleting an old output B shortens later prompts but breaks the cache match for the following suffix S. In this illustrative fixed-action example, B = 1,000, S = 5,000 and c = 0.1. Ten remaining requests save 1,000 fresh-token cost units but incur 4,500 units to rebuild the suffix; breaking even requires 45 requests. Prompt lengths are schematic, and patch quality is evaluated separately.
  • Behavioral Compensation: Agents are not static. If you remove context to save money, the agent may become "confused" and take more steps to find the information it lost, ultimately resulting in a higher bill.

  • The "Sign Flip": A context management method that looks like a winner at full price ($c=1$) can become a massive loser when a 90% cache discount is applied ($c=0.1$).
  • Concurrent Bias: The authors discovered that running multiple experiments side-by-side on the same server can lead to "prefix warming," where one experiment's cache hits are actually powered by a different experiment's previous requests. This can lead to misleadingly low cost reports if not carefully isolated.

Key Results

  • Retrieval is a double-edged sword: Adding "evidence packages" (retrieved code) reduced the number of search commands by 18–34%. However, it increased total costs by 33–42% at a tenth-price cache, with no measurable improvement in how many bugs the agent actually fixed.
  • Masking can backfire: A common strategy of "rolling masking" (keeping only the last 10 observations) saved 19–36% at full price but increased costs by 38–116% at a tenth-price cache.
  • Break-even Masking: The authors proposed a policy that only masks observations when the predicted future savings ($R \cdot c \cdot B$) outweigh the immediate cost of recomputing the suffix ($(1-c) \cdot S$). This was the only method that remained cost-competitive across different pricing tiers.
Figure 3: Cost of three masking policies relative to no masking as the price of a cached token varies, on 201 fresh Verified tasks in mini-SWE-agent. Rolling mask- ing doubles cost at c = 0.1 and saves at c = 1; break- even masking, set for c = 0.1, stays near no masking when caching is cheap and saves when it is not.

Limitations and Future Work

  • Model Scope: The study focused on open-weights models served locally (vLLM). While these models are powerful, the behavior of "frontier" closed models (like GPT-4o or Claude 3.5) under similar constraints remains an open question.
  • Fixed Budgets: The experiments used bounded output and step limits. In real-world scenarios where agents might run for hundreds of steps, the cost dynamics of the "long tail" of a run become even more critical.
  • Future Directions: The authors suggest that future research should report uncached, cached, and output tokens separately and evaluate context management alongside agent "stopping" behavior (when the agent decides it's done).

Summary updated

Abstract

Coding agents repeatedly send their conversation history to a language model. Adding useful code may reduce search, while removing old tool outputs may shorten later prompts. But do these changes reduce the cost of solving a task? We study this as a measurement problem in more than 20,000 runs of open models on SWE-bench Verified, SWE-bench and Loc-Bench. We account separately for uncached input, cached input and output tokens. Three findings emerge. First, in our main evidence-package experiments, retrieved code reduces read and search commands by 18--34%, but increases cost by 33--42% when cached input costs one tenth of uncached input, without a measurable improvement in resolution. Second, removing old observations can either save money or increase cost: the same approach saves 19--36% without a cache discount but increases cost by 38--116% with a tenth-price cache. Third, shorter prompts do not guarantee cheaper runs because agents may respond by taking more steps. We also find that concurrent experiments can share cached prefixes and distort cost comparisons. These results show why context-management methods should be evaluated using both token prices and the agent's subsequent behavior.
BibTeX
@misc{yang2026context,
  author = {Zerui Yang and Miaomiao Dong and Quan Xue and Chaorui Zhang and Xuanyu Zheng and Xiyuan Wang and Haolin Ruan and Xueyan Niu and Guangcheng Song and Weixi Zhang and Wei Han and Bo Bai},
  title = {{Context Has a Price: Adding and Removing Context for LLM Coding Agents under Prefix Caching}},
  year = {2026},
  eprint = {2611.22238},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22238},
}
Figure