SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification

Yulu An, Yifei Feng, Xiangyang YinarXiv 2026 · arXiv

TL;DR

SpecCompact is a new framework that drastically speeds up how LLM agents summarize their own interaction histories—a process called context compaction. By generating different parts of a summary (like "goals" and "next steps") in parallel and using a "semantic" verification method that focuses on meaning rather than exact wording, SpecCompact achieves up to a 5.6x speedup in benchmarks and an 8.7x median speedup in real-world user studies.

What Problem Does This Paper Address?

As LLM agents (like coding assistants or personal AI) perform long-running tasks, their conversation history grows massive. To stay within context window limits and maintain speed, these agents must periodically "compact" their history into a concise summary.

However, using a powerful, large model to generate these summaries token-by-token is slow, creating a frustrating "pause" in the user experience. While standard speculative decoding (using a small model to draft tokens for a large model to verify) can help, it usually requires an exact match between the models. In summarization, many different wordings can be equally "correct," meaning standard speculative decoding leaves a lot of potential speed on the table.

Methods

SpecCompact introduces a two-pronged approach to break this bottleneck: Slot-Parallelism and Semantic Speculative Verification.

  1. Slot-Parallelism: The authors observed that structured summaries are often composed of independent "slots" (e.g., Goal, Constraints, Progress, Next Steps). Crucially, they found that these slots can be generated simultaneously without needing to see each other's output.
  2. Semantic Speculation: Instead of verifying every single token, SpecCompact uses a small "drafter" model (Qwen3-0.6B) to propose entire lines. A larger "target" model (like Gemma4-31B or Qwen3.5-122B) then performs a binary check: Is this line factually supported by the history and consistent with what we've already accepted? If yes, it's accepted; if no, the target model quickly repairs it.
Figure 3: Overview of SpecCompact.

To make this efficient, the authors implemented several system optimizations, including shared-prefix caching (so the target model doesn't re-process the long history for every single slot) and draft buffering.

Execution optimizations including independent scheduling and shared-prefix caching.

Datasets

The researchers trained and evaluated SpecCompact using several rigorous benchmarks:

  • AppWorld & $\tau^3$: Benchmarks for evaluating LLM agents on complex, multi-step tasks like coding and tool use.
  • LiveClawBench: Used for preliminary studies on summary adequacy.
  • LongBench & RULER: Standard benchmarks for testing how well models handle and retrieve information from very long contexts (up to 16k tokens in this study).

Key Insights and Ideas

  • The Independence of Summary Slots: Perhaps the most surprising finding is that generating summary fields in parallel (e.g., "Next Steps" before "Goal") has almost zero impact on the quality of the information retained. This breaks the sequential dependency that usually slows down LLM generation. Comparison of monolithic vs. independent slot generation showing minimal quality loss.

  • Semantic > Token-Level: By relaxing the requirement for an exact token match, the system can leverage the speed of the small drafter much more effectively. If the drafter gets the point right, the target model moves on.

  • System Bottlenecks Matter: Parallelism alone isn't a silver bullet. Without shared-prefix caching, the overhead of the target model re-reading the long history would negate the gains from parallel drafting.
Figure 2: Slot structure and independent generation. (a) Fields in OpenClaw and Hermes compaction templates. (b) Reference adequacy over 100 trajectories on LiveClawBench (Long et al., 2026), with six slots per trajectory.

Key Results

SpecCompact delivers significant efficiency gains with manageable trade-offs in agent performance:

  • Speedups: It achieved 2.04x to 5.63x speedups over standard summarization methods across various agent workloads.
  • User Experience: In a native deployment with the Hermes agent, users experienced a 8.68x median speedup in compaction latency.
  • Task Success: While there is a slight "accuracy tax" in some scenarios (e.g., a small dip in success for Gemma4), in other cases, like with Qwen3.5, SpecCompact actually improved task completion rates compared to the baseline.
Figure 1: Task success and compaction speedup with Gemma4-31B-IT, measured in separate evaluations.

Limitations and Future Work

  • Accuracy Consistency: The authors note that SpecCompact doesn't yet offer a consistent accuracy advantage over standard summarization; it's primarily a speed-oriented framework.
  • Drafter Alignment: Future work could involve using Reinforcement Learning (RL) to better align the small drafter's output with the target model's preferences, potentially reducing the number of "repairs" needed.
  • Attention Optimization: The team noticed that different slots look at different parts of the history (e.g., "Goals" are usually at the start). They suggest "slot-conditioned sliding-window attention" as a future path to speed up the drafter even further.

Summary updated

Abstract

LLM agents use context compaction to manage growing interaction histories, but summarizing long histories with large models can introduce substantial delays. Compaction therefore requires balancing information retention against latency. We observe that structured summary slots can retain information when generated independently. Based on this observation, we present SpecCompact, a framework that accelerates compaction through slot-parallel semantic speculation. SpecCompact fine-tunes a small drafter and generates slots in parallel. The target model verifies each proposed line and repairs rejected content, while subsequent drafting uses the updated slot prefix. System optimizations further reduce repeated context processing and model calls. Across AppWorld and $τ^3$ benchmarks with two target models, SpecCompact achieves $2.04$--$5.63\times$ compaction speedups over Hermes-style summarization. Separate downstream evaluations reveal model- and task-dependent performance trade-offs. A user study within Hermes further shows an $8.68\times$ median compaction speedup.
BibTeX
@misc{an2026speccompact,
  author = {Yulu An and Yifei Feng and Xiangyang Yin},
  title = {{SpecCompact: Accelerating LLM Agent Context Compaction with Slot-Parallel Semantic Speculative Verification}},
  year = {2026},
  eprint = {2611.06876},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06876},
}
Figure