CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks

Jiayi Chen, Lizhe Chen, Yujia Chen, Shihan Dou, Ming Zhang, Chendi Ge, Shiguang Ni, Tao Gui, Qi Zhang, Xuanjing Huang, Maxm PanarXiv 2026 · arXiv

As Large Language Model (LLM) agents move from simple chat interfaces to long-horizon tasks—like debugging a massive codebase or conducting deep web research—they inevitably hit a wall: the context window. To keep going, these agents must "compact" their history, turning thousands of tokens of tool outputs and reasoning into a concise summary.

But does a summary that looks good to a human actually preserve the information an agent needs to finish the job? A new paper from researchers at Fudan University, Tencent, and Tsinghua University introduces CompAct-Bench to find out.

TL;DR

CompAct-Bench is a new benchmark that evaluates LLMs not on how well they summarize text, but on how well their "compacted" context supports an agent in actually completing a task. The results are a wake-up call: even frontier models like Claude Opus 5 lose significant functional information during compaction, with average task success rates dropping to just 56.7% after history is compressed.

What Problem Does This Paper Address?

When an agent’s interaction history approaches the model's context limit, developers usually choose between truncation (deleting old messages) or compaction (summarizing the history). Compaction is the preferred "smart" choice, but it’s historically been evaluated using standard NLP metrics like ROUGE or human preference, which don't measure utility.

An agent needs more than a gist; it needs the "working state"—which hypotheses were ruled out, which files were already edited, and what the specific next step was. If a compactor omits a "tentative" flag or a specific identifier, the agent might repeat hours of work or confidently hallucinate a conclusion based on incomplete data.

Methods

The researchers developed a "process-level" evaluation pipeline that turns successful agent trajectories into compaction tests.

  1. Trajectory Collection: They first collect trajectories where a "gold" agent successfully completes a task.
  2. Boundary Insertion: They "pause" the agent at 25%, 50%, or 75% of its progress.
  3. Compaction: The model under evaluation (the "compactor") is asked to compress the history before that boundary into a much smaller representation (usually 10% of the original length).
  4. Execution: A fixed, strong executor resumes the task using only the compacted summary and the current environment state.
  5. IRC (Iterative Ratio Calibration): To ensure a fair fight, they use a novel IRC procedure. If a model produces a summary that is too long or too short, the system provides feedback and asks it to try again until it hits the target retention ratio.
Overview of the CompAct-Bench pipeline: from successful trajectory collection to post-compaction execution.

Datasets

CompAct-Bench spans 200 tasks across three distinct domains:

  • BrowseComp (79 tasks): Multi-step web information seeking.
  • SWE-bench (96 tasks): Real-world software engineering issues.
  • CompanyBench (25 tasks): An internal suite for workspace automation (files, tools, and persistent states).

Key Insights and Ideas

  • The "State Location" Divergence: Compaction performance behaves differently depending on where the "truth" lives. In Coding, performance actually improves at later boundaries. Why? Because the progress is "materialized" in the code files themselves. In Search, performance drops as the task goes on because the "evidence" only exists in the conversation history.
  • Omission vs. Distortion: Failures aren't just about forgetting. "Factual Distortion" and "Premature Commitment" are major killers. A compactor might turn a "possible lead" into a "confirmed fact," causing the agent to terminate early with a wrong answer.
  • Redundant Work: When critical omissions occur, agents don't just fail; they get stuck in loops, re-running tools to find information they technically already had, eventually exhausting their action budget.
Error prevalence analysis showing how different compaction errors (like premature commitment) lead to different failure modes.

Key Results

The study evaluated 12 frontier models, including the Claude, GPT, Gemini, and Qwen families.

  • Compaction is highly lossy: While the executor solved 100% of these tasks with full history, the average post-compaction accuracy was only 56.7%.
  • Top Performers: Claude Opus 5 led the pack with 68.5% overall accuracy, followed by Gemini 3.8 Flash at 65.5%.
  • The 10% Wall: Compressing history to 10% of its size is a massive hurdle. Even the best models struggle to decide what is "load-bearing" information versus "noise."
Accuracy trends across execution stages. Note how Coding (purple) improves over time while Search (blue) degrades.

Limitations and Future Work

The authors acknowledge that their 10% retention target is aggressive and might be too tight for some complex tasks. Furthermore, the benchmark currently uses a "fixed" executor; future iterations could explore how different executor-compactor pairings interact.

The most promising direction for future work is "prospective compaction"—training models specifically to summarize for future action rather than retrospective reporting. CompAct-Bench provides the first rigorous framework to measure progress toward that goal.

Summary updated

Abstract

As language-model agents are applied to increasingly extended tasks, their working context continuously accumulates interaction history, tool observations, intermediate decisions, and task progress. Context compaction offers a practical way to reduce this growing history while allowing execution to continue, yet the compaction ability of modern LLMs remains insufficiently understood. We introduce CompAct-Bench, a benchmark for evaluating working-context compaction through its downstream effect on agent task completion. Starting from successful agent trajectories, we insert compaction boundaries at different stages of execution, compress the preceding working context with the model under evaluation, and let the same executor continue the original task. To enable fair comparison across compactors, we evaluate them under controlled retention ratios using a lightweight iterative ratio-adjustment procedure. CompAct-Bench covers information-seeking, software-engineering, and workspace-oriented tasks through BrowseComp, SWE-bench, and an internal CompanyBench evaluation. Our experiments show that working-context compaction remains highly lossy even for frontier models: Claude- and GPT-family models perform strongest overall, yet substantial degradation in downstream task accuracy persists under aggressive compaction. Further analysis reveals that failures are often caused not only by omitted information, but also by distorted task state, such as turning unresolved issues into apparent conclusions or forgetting previously established results, which can lead to incorrect decisions or redundant execution. These findings highlight working-context compaction as a distinct and still challenging ability for LLMs operating in extended agent tasks.
BibTeX
@misc{chen2026compact,
  author = {Jiayi Chen and Lizhe Chen and Yujia Chen and Shihan Dou and Ming Zhang and Chendi Ge and Shiguang Ni and Tao Gui and Qi Zhang and Xuanjing Huang and Maxm Pan},
  title = {{CompAct-Bench: A Benchmark for Working-Context Compaction in Long-Horizon Agent Tasks}},
  year = {2026},
  eprint = {2611.25673},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25673},
}
Figure