Adversarial Speculative Reasoning Improves Long Horizon Agents

Hongyin Luo, Wei Fang, Lijie Yang, Owen Stepan, Daniel Suit, Jeff Krapf, Dana Wensberg, Jack O'BrienarXiv 2026 · arXiv

TL;DR

Speculative decoding is traditionally viewed as a "lossless" speed optimization, but this paper reveals it actually introduces structured numerical noise that degrades long-horizon reasoning. By training draft models to be adversarial—intentionally failing at critical decision points—the authors force the target model to process key reasoning steps with maximum numerical precision, boosting agent success rates by up to 7% with negligible impact on speed.

What Problem Does This Paper Address?

In the world of Large Language Models (LLMs), speculative decoding is a standard trick for speed: a tiny "draft" model guesses a sequence of tokens, and a massive "target" model verifies them in one go. Mathematically, this is guaranteed to produce the same output distribution as the target model alone.

However, the authors discover that in real-world deployments—using low-precision weights (FP4/FP8), quantized KV caches, and sparse attention—this "lossless" guarantee is a myth. They found that numerical noise accumulates within the verification window. As shown in Figure 1, the further a token sits in a verification block, the noisier its computation becomes.

Figure 1. Single-pass numerical noise |∆log p| by verify slot between repeated forwards on an identical prefix. (a) Mean deviation, with the per-slot slope. (b, c) Fraction of tokens shifted by more than 1 and 2 nats (log scale). (d) Mean deviation relative to slot 0, with least-squares fits: both models grow by about 4% of their slot-0 level per slot. Slopes with 95% bootstrap CIs over positions: +0.0195 [+0.0174, +0.0214] (GLM-5.3) and +0.0094 [+0.0078, +0.0110] (DeepSeek-V4.1-Flash).

For "long-horizon" agents (like AI software engineers that run for hundreds of steps), this tiny per-step noise compounds. After about 30 turns, the noise significantly alters the model's behavior, often leading to reasoning failures in complex tasks.

Methods

The researchers propose a counter-intuitive solution: Adversarial Speculative Reasoning. Instead of training the draft model to perfectly mimic the target, they train it to fail at the most important moments.

  1. Identifying Decision Points: They use the target model as a judge to find "committing sentences"—critical moments where the agent decides on a strategy, interprets a requirement, or chooses whether to run a test.
  2. Adversarial Training: They use Direct Preference Optimization (DPO) to train the draft model to propose the undesirable path at these decision points.
  3. Precision Re-allocation: Because the draft proposes a "bad" token at a critical juncture, the target model rejects it. This triggers a "restart," forcing the target model to resample that specific token at Slot 0—the most numerically precise position in a verification block.

Essentially, they use the draft model as a "precision allocator," ensuring that routine text is accelerated while critical reasoning is handled with the highest possible fidelity.

Datasets

The authors evaluated their approach on two demanding long-horizon agentic benchmarks:

  • DeepSWE: A benchmark consisting of 113 complex software engineering tasks where an agent must resolve real-world GitHub issues.
  • NL2Repo-Bench: A task requiring agents to build multi-module Python libraries from scratch based on requirements.

Training data was generated using "fork pairs"—taking real agent traces and resampling multiple continuations at decision points to create preferred vs. adversarial examples.

Key Insights and Ideas

  • Speculative Decoding is an Active Allocator: The draft model isn't just an accelerator; it's a tool to control where rejections happen. Since rejections reset the noise floor, we can strategically fail to improve accuracy.
  • The "Slot 0" Advantage: The first token in a verification block is as precise as standard single-token decoding. Every subsequent slot is ~4% noisier.
  • Noise Surfaces Downstream: As seen in Figure 2, the discrepancy between "precise" decoding and "noisy" verification-written context only becomes significant after a long horizon (30+ turns), explaining why this hasn't been a major issue for short-chat models.
Long-range deviations between verify-written and decode-written context become significant after 30 turns.
  • Tiny Training, Big Impact: The authors achieved these gains by training only 0.06% to 0.28% of the total system parameters (just parts of the draft model), leaving the massive target model completely untouched.

Key Results

The adversarial approach yielded significant improvements in agent performance without changing the target model's weights:

  • GLM-5.3-nvfp4: On DeepSWE, the success rate jumped from 60.6% to 67.6% (+7.0 points).
  • DeepSeek-V4.1-Flash: Gained +4.0 points on DeepSWE and +1.4 points on NL2Repo-Bench.
  • Efficiency: The "cost" of these rejections was minimal. Decoding speed only decreased by about 2%, a negligible trade-off for the accuracy gains.
  • Direction Matters: Training the draft to be "helpful" (agreeing with the target at decision points) actually hurt performance, confirming that the benefit comes specifically from forcing rejections at critical moments.
Comparison of pass rates across different draft configurations.

Limitations and Future Work

  • Decision Point Detection: The current method relies on a "judge" to find committing sentences. Generalizing this to non-coding tasks or different reasoning styles remains an open challenge.
  • Hardware Determinism: If future hardware and kernels become perfectly bitwise-deterministic, the specific numerical noise benefit might diminish. However, the authors argue that as long as we use low-precision quantization to scale, some level of non-determinism is likely to persist.
  • Draft Capacity: The draft model must be capable of learning the "framing" of a decision. As shown in Figure 4, while drafts can learn how a model thinks, they aren't strong enough to learn semantic correctness (right vs. wrong), which is why the adversarial "rejection" strategy is necessary.
Figure 4. Held-out preference accuracy of trained drafts on tasks unseen in training, best configuration per preference and direction; error bars are 95% binomial confidence intervals, the dashed line is chance. Top: the two preferences we deploy. Bottom: two reference preferences we do not use. For thinking length the preferred side is the shorter thinking; for right vs. wrong it is the right decision, and the bars shown are the best of 36 (forward) and 12 (adversarial) configurations whose medians are 0.52 and 0.46.

Summary updated

Abstract

Speculative decoding has been treated purely as a systems optimization that speeds up large language model decoding without changing the model behavior. Under this view, the growing effort to train better draft models has been focused solely on acceptance rate. We argue that this view does not hold true for long-horizon agent reasoning. Measuring frontier open-weight models on complex agentic tasks with different speculative decoding approaches, we found that numerical noise grows within verification windows, and the draft acceptance results of the thinking process can change the output distribution of the model after 30 steps, thus noticeably influencing the reasoning accuracy of long-horizon agents in random directions. In this work, we improve the reasoning capability of target models by controlling such noise. While we cannot change the overall token distribution verified by the target model, we can encode critical reasoning with highest precision by training the draft model to be rejected. We identify these points in agent traces and train the draft model to favor adversarial preference. At inference the trained draft and the target part ways precisely at these decision points, verification restarts, so that critical thinking tokens are sampled from the target model and encoded under its least noisy conditions. This approach makes critical reasoning better indexed and more durable over a long horizon. On complex agentic coding benchmarks, our approach raises end-to-end success rate with no change to the target model or its output distribution. Our results recast the speculative decoding mechanism from a pure accelerator into an active allocator for context precision that improves the accuracy of long-horizon agents by training 0.06% to 0.28% of hosted parameters.
BibTeX
@misc{luo2026adversarial,
  author = {Hongyin Luo and Wei Fang and Lijie Yang and Owen Stepan and Daniel Suit and Jeff Krapf and Dana Wensberg and Jack O'Brien},
  title = {{Adversarial Speculative Reasoning Improves Long Horizon Agents}},
  year = {2026},
  eprint = {2611.05345},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05345},
}
Figure