TraceSpec: Understanding and Improving Speculative Decoding for LLM Agents

Jihao Zhao, Shuaishuai Zu, Pengfei Li, Biao Qin, Guanghuan Fang, Ben Feng, Baochuan YangarXiv 2026 · arXiv

TL;DR

TraceSpec is a new framework designed to optimize speculative decoding specifically for LLM agents by analyzing their execution trajectories. By using "state-conditioned" adaptive training—prioritizing the specific phases and context lengths where draft models struggle most—TraceSpec achieves throughput gains of up to 22.6% over strong baselines like DSpark.

What Problem Does This Paper Address?

Speculative decoding (SD) is a popular technique to speed up Large Language Model (LLM) inference. It works by having a small, fast "draft" model guess the next few tokens, which a large "target" model then verifies in parallel. If the draft is right, you get multiple tokens for the price of one verification round.

However, LLM agents (like those that browse the web or write code) present a unique challenge. Their workloads involve long, evolving histories, tool calls, and distinct reasoning phases (planning, acting, reflecting). Standard SD metrics are usually "aggregate"—they average performance across an entire session. This hides critical failures: a draft model might be great at generating conversational text but terrible at predicting the specific syntax of a tool call when the context window hits 32k tokens. The authors argue that to truly speed up agents, we need to understand where and why the draft model fails along a trajectory.

Methods

The authors introduce TraceSpec, a framework that connects trajectory evaluation with adaptive draft training. Key components include:

  1. Trajectory Diagnosis: They use a "fixed-prefix replay" method to compare different draft models on identical agent states. This allows them to isolate how context length, task phase, and workload impact the "acceptance rate" (how many tokens the draft gets right).
  2. Adaptive Supervision: Instead of training the draft model only on final assistant answers, TraceSpec supervises every intermediate turn in a trajectory.
  3. State-Conditioned Allocation: Not all training data is equal. TraceSpec calculates an "acceptance gap" for different states (e.g., "Software Engineering + Action Phase + 8k Context"). It then adaptively allocates a finite training budget to the states where the draft model's disagreement with the target model is highest.
  4. Event-Normalized Sampling: To prevent segments with many tokens from hogging the budget, they normalize sampling so that the model visits "hard" states more frequently but stays within a total prediction budget.
Figure 1: TraceSpec overview. (a) Trajectory-state construction, coverage, and source-task splits. (b) Dynamic execution and fixed-prefix replay with trajectory diagnosis. (c) Full-turn supervision with state-conditioned allocation and event-normalized sampling under a fixed prediction budget.

Datasets

The study introduces TraceSpec-Traj, a specialized corpus for agent research:

  • Training Set: 10,000 states derived from 5,795 source tasks.
  • Test Set: 2,500 states from 1,380 disjoint tasks.
  • Domains: The data spans web browsing (from DeepResearch-traj) and software engineering (from Nebius SWE-agent and R2E-Gym).
  • Evaluation Benchmarks: The models were tested on BrowseComp-Plus and SWE-bench Verified.

Key Insights and Ideas

  • The Context Length Paradox: Interestingly, aggregate acceptance rates often appear to increase as the context gets longer. However, the authors discovered this is a mirage. When they looked at "matched late actions" (comparing the same type of action at different context lengths), acceptance actually decreased. The aggregate rise was simply because the model spent more time on easier, repetitive reasoning steps in longer contexts.
  • Phase-Specific Difficulty: A draft model’s accuracy varies wildly depending on what the agent is doing. For example, in web research, "Action" phases often have higher acceptance than "Planning" phases.
  • Workload Matters: Browsing and software engineering tasks have completely different acceptance profiles. A drafter optimized for one may struggle with the other, necessitating the state-conditioned approach.
Figure 2: Baseline replay acceptance by context length. Qwen3-8B on TraceSpec-Traj, grouped by domain, tempera- ture, and actual input length. All methods receive the same prefixes.

Key Results

  • Throughput Gains: TraceSpec improved generation throughput by 12.5% to 22.6% over the DSpark baseline across various target model sizes (Qwen3-4B, 8B, 14B).
  • Superior Acceptance: In phase-specific testing, TraceSpec led in 68 out of 72 comparisons against five other speculative decoding baselines (including EAGLE-3 and DFlash).
  • Ablation Success: The adaptive budget allocation alone increased the accepted token length by 3.3% to 10.0% compared to standard fixed-turn supervision, proving that where you train is just as important as what you train on.

Limitations and Future Work

  • Budget Sensitivity: The framework relies on a pre-defined "prediction budget." The authors note that the optimal size of this budget for different scales of models remains to be fully explored.
  • Architecture Locking: While highly effective, the current implementation is built on the DSpark architecture. Future work could involve applying these trajectory-aware training strategies to other speculative decoding architectures like EAGLE or multi-token predictors.
  • Online Adaptation: Currently, the draft model is trained offline on collected trajectories. A promising next step is "online" adaptation, where the draft model learns to improve its performance in real-time as an agent navigates a new, unseen environment.

Summary updated

Abstract

LLM agents execute tasks through repeated model calls over evolving histories of reasoning, actions, and tool feedback. Aggregate speculative-decoding metrics merge these distinct decisions, providing limited guidance on where draft supervision is needed. We present TraceSpec, a framework connecting trajectory evaluation with adaptive draft training. It combines dynamic execution with fixed-prefix replay and introduces TraceSpec-Traj, a corpus of 10,000 training states and 2,500 test states spanning browsing and software engineering. Analysis of five speculative baselines reveals that aggregate acceptance can rise with context length while declining within matched late browsing actions. Workload and phase further distinguish these trends, motivating supervision guided by trajectory conditions. TraceSpec trains on intermediate assistant turns and allocates a fixed prediction budget using state-conditioned estimates of prefix-aware acceptance gaps. Across three target sizes and two temperatures, it improves replay throughput by 12.5% to 22.6% over its DSpark base. Qwen3-4B ablations show that adaptive allocation increases accepted length by 3.3% to 10.0% over fixed full-turn supervision, with additional comparisons substantiating the roles of state conditioning and consecutive-token agreement. The resulting evaluation resources, trajectory findings, and training strategy connect the analysis of agent execution to practical draft improvement.
BibTeX
@misc{zhao2026tracespec,
  author = {Jihao Zhao and Shuaishuai Zu and Pengfei Li and Biao Qin and Guanghuan Fang and Ben Feng and Baochuan Yang},
  title = {{TraceSpec: Understanding and Improving Speculative Decoding for LLM Agents}},
  year = {2026},
  eprint = {2610.41177},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.41177},
}
Figure