CarryOn: Learning What to Keep in Context for Long-Horizon LLM Agents

Lihuan Li, Ruiyi Yang, Jiru Wang, Yifan Duan, Xiachong Lin, Hao Xue, Yang Song, Flora SalimarXiv 2026 · arXiv

TL;DR

CarryOn is a lightweight context manager for long-horizon LLM agents that separates high-level progress tracking from fine-grained evidence selection. By using a learned selector to retrieve specific "receipts" from history only when they are relevant, it achieves a 75.7% task success rate—outperforming the strongest baselines by 6.7 percentage points while significantly reducing token costs.

What Problem Does This Paper Address?

As LLM agents tackle increasingly complex, multi-step tasks (like managing an office workflow or resolving a customer support ticket), their interaction histories balloon. This creates a "context management" dilemma.

If you keep the Full History, the input cost skyrockets, and the model may get distracted by irrelevant noise. If you Summarize or Prune, you risk losing the "exact details"—like a specific tracking number or a coworker's email—that were mentioned ten steps ago but are suddenly critical for the final action.

The authors identify that shorter contexts don't automatically lead to better decisions. The challenge is keeping the agent's "working memory" concise regarding task progress while ensuring the specific evidence needed for the current action remains within reach.

Methods

The core innovation of CarryOn is the separation of concerns, much like a traveler using a small "carry-on" bag for essentials while keeping the rest of their luggage accessible if needed.

Figure 1: CarryOn: carry what matters. Left: Context-management challenges and CarryOn’s separation of compact task state from fine-grained evidence. Right: Success–efficiency comparison (top) and task-success gains over the strongest baseline, averaged by benchmark or Actor (bottom).

The system consists of three main components:

  1. Compact Control State: Instead of the full history, the "Actor" (the LLM doing the work) maintains a bounded "MEMORY" field (typically 128 tokens). This tracks what is done, what is open, and the next intended step.
  2. Fine-grained Evidence Units: Original observations are split into small, source-linked units. These are stored in an archive rather than being summarized into a lossy paragraph.
  3. Learned Selector & Adaptive Retention: A lightweight reranker (based on Qwen3-Reranker-0.6B) scores these units based on the current task and the latest observation.
  4. Relevance Ranking: It pulls back old evidence that becomes relevant again.
  5. Temporary Evidence Expansion: It predicts which units will be needed for the immediately next action and includes them even if they exceed the base budget.
  6. Adaptive Retention: It always keeps the most recent interaction to maintain immediate continuity.
The CarryOn architecture: History, Evidence Selection, and Acting.

Datasets

The authors evaluated CarryOn across three diverse long-horizon benchmarks:

  • AppWorld: Complex digital tasks involving multiple apps (e.g., Venmo, contacts, calendar).
  • OfficeBench: Realistic office automation workflows.
  • Tau2 Telecom: Conversational customer support scenarios.

They tested the system using three different "Actor" models: Claude 3.5 Sonnet, Qwen3.8-27B, and Gemma-4-31B, demonstrating that the context manager generalizes across different LLMs.

Key Insights and Ideas

  • "Evidence Reselection" is Critical: In 40.2% of successful task runs, CarryOn brought back evidence that had been previously omitted from the context. This proves that "forgetting" and then "remembering" is a more efficient strategy than trying to keep everything in view at once.
  • State vs. Evidence: The paper argues that an agent needs two different types of memory: a semantic understanding of progress (State) and literal fragments of data (Evidence). Mixing them into a single summary often degrades both.
  • Efficiency Metrics Matter: The authors introduce $S_\alpha$ (Success-efficiency score) and NCS (Normalized Cost per Success). These metrics prevent "brute-forcing" success by penalizing agents that succeed but use an excessive number of tokens or API calls.
  • Cross-Actor Generalization: Interestingly, a selector trained using data from one model (e.g., Claude) effectively manages context for entirely different models (e.g., Qwen or Gemma) without retraining.

Key Results

CarryOn consistently outperformed existing methods like LLMLingua-2 (compression) and AGORA (history selection).

  • Task Success: Achieved 75.7% macro-averaged success, beating the strongest baseline (SWE-Pruner) by 6.7 percentage points and Full History by 13.3 points.
  • Resource Efficiency: It achieved the lowest Normalized Cost per Success (NCS) at 0.792, meaning it is roughly 20% more cost-effective per successful task than using the full history.
  • Trajectory Length: CarryOn completed tasks with 10.1% fewer tool calls than the Full History baseline, suggesting that a cleaner context leads to more direct, less "confused" agent behavior.

Limitations and Future Work

  • Text-Only Focus: The current implementation is designed for text-based interactions. Extending this to visual or multimodal observations (e.g., screenshots of a UI) would require new ways to represent and "split" evidence units.
  • Domain Specificity: While the selector generalizes across models, the authors currently train a separate selector for each benchmark. Future work could explore a more universal selector.
  • Privacy/Security: Retaining fine-grained original observations in an archive raises potential data privacy concerns that would need robust access control in production environments.

Summary updated

Abstract

Long-horizon LLM agents can coordinate tools and applications to complete complex, multi-step tasks. As their interaction histories grow, effective context management becomes essential for reliable and efficient execution. Existing methods reduce context through summarization, pruning, or history selection. However, shorter contexts do not necessarily support better decisions: summaries can omit exact details needed for later actions, while retaining entire interactions can carry substantial irrelevant information. We propose CarryOn, a lightweight context manager that pairs compact execution state with fine-grained, source-linked evidence. A learned selector and adaptive retention policy keep relevant evidence available without additional LLM calls for context management. We introduce two complementary evaluation metrics: $S_α$ prioritizes task success while accounting for resource use, and normalized cost per success (NCS) measures the resources spent per successful task. Across AppWorld, OfficeBench, and Tau2 Telecom with three LLM Actors, CarryOn achieves 75.7% macro-averaged task success, exceeding the strongest baseline by 6.7 percentage points. It also achieves the best average scores on both metrics.
BibTeX
@misc{li2026carryon,
  author = {Lihuan Li and Ruiyi Yang and Jiru Wang and Yifan Duan and Xiachong Lin and Hao Xue and Yang Song and Flora Salim},
  title = {{CarryOn: Learning What to Keep in Context for Long-Horizon LLM Agents}},
  year = {2026},
  eprint = {2611.17808},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17808},
}
Figure