As Large Language Model (LLM) agents move from simple chat interfaces to long-horizon tasks—like debugging a massive codebase or conducting deep web research—they inevitably hit a wall: the context window. To keep going, these agents must "compact" their history, turning thousands of tokens of tool outputs and reasoning into a concise summary.
But does a summary that looks good to a human actually preserve the information an agent needs to finish the job? A new paper from researchers at Fudan University, Tencent, and Tsinghua University introduces CompAct-Bench to find out.
TL;DR
CompAct-Bench is a new benchmark that evaluates LLMs not on how well they summarize text, but on how well their "compacted" context supports an agent in actually completing a task. The results are a wake-up call: even frontier models like Claude Opus 5 lose significant functional information during compaction, with average task success rates dropping to just 56.7% after history is compressed.
What Problem Does This Paper Address?
When an agent’s interaction history approaches the model's context limit, developers usually choose between truncation (deleting old messages) or compaction (summarizing the history). Compaction is the preferred "smart" choice, but it’s historically been evaluated using standard NLP metrics like ROUGE or human preference, which don't measure utility.
An agent needs more than a gist; it needs the "working state"—which hypotheses were ruled out, which files were already edited, and what the specific next step was. If a compactor omits a "tentative" flag or a specific identifier, the agent might repeat hours of work or confidently hallucinate a conclusion based on incomplete data.
Methods
The researchers developed a "process-level" evaluation pipeline that turns successful agent trajectories into compaction tests.
- Trajectory Collection: They first collect trajectories where a "gold" agent successfully completes a task.
- Boundary Insertion: They "pause" the agent at 25%, 50%, or 75% of its progress.
- Compaction: The model under evaluation (the "compactor") is asked to compress the history before that boundary into a much smaller representation (usually 10% of the original length).
- Execution: A fixed, strong executor resumes the task using only the compacted summary and the current environment state.
- IRC (Iterative Ratio Calibration): To ensure a fair fight, they use a novel IRC procedure. If a model produces a summary that is too long or too short, the system provides feedback and asks it to try again until it hits the target retention ratio.
Datasets
CompAct-Bench spans 200 tasks across three distinct domains:
- BrowseComp (79 tasks): Multi-step web information seeking.
- SWE-bench (96 tasks): Real-world software engineering issues.
- CompanyBench (25 tasks): An internal suite for workspace automation (files, tools, and persistent states).
Key Insights and Ideas
- The "State Location" Divergence: Compaction performance behaves differently depending on where the "truth" lives. In Coding, performance actually improves at later boundaries. Why? Because the progress is "materialized" in the code files themselves. In Search, performance drops as the task goes on because the "evidence" only exists in the conversation history.
- Omission vs. Distortion: Failures aren't just about forgetting. "Factual Distortion" and "Premature Commitment" are major killers. A compactor might turn a "possible lead" into a "confirmed fact," causing the agent to terminate early with a wrong answer.
- Redundant Work: When critical omissions occur, agents don't just fail; they get stuck in loops, re-running tools to find information they technically already had, eventually exhausting their action budget.
Key Results
The study evaluated 12 frontier models, including the Claude, GPT, Gemini, and Qwen families.
- Compaction is highly lossy: While the executor solved 100% of these tasks with full history, the average post-compaction accuracy was only 56.7%.
- Top Performers: Claude Opus 5 led the pack with 68.5% overall accuracy, followed by Gemini 3.8 Flash at 65.5%.
- The 10% Wall: Compressing history to 10% of its size is a massive hurdle. Even the best models struggle to decide what is "load-bearing" information versus "noise."
Limitations and Future Work
The authors acknowledge that their 10% retention target is aggressive and might be too tight for some complex tasks. Furthermore, the benchmark currently uses a "fixed" executor; future iterations could explore how different executor-compactor pairings interact.
The most promising direction for future work is "prospective compaction"—training models specifically to summarize for future action rather than retrospective reporting. CompAct-Bench provides the first rigorous framework to measure progress toward that goal.
Summary updated