When RL Meets Adaptive Speculative Training: A Unified Training-Serving System

Junxiong Wang, Fengxiang Bie, Jisen Li, Zhongzhu Zhou, Zelei Shao, Yubo Wang, Yinghui Liu, Qingyang Wu, Avner May, Sri Yanamandra, Yineng Zhang, Ce Zhang, Tri Dao, Percy Liang, Ben Athiwaratkun, Shuaiwen Leon Song, Chen Tang, Xiaoxia Wu

TL;DR

Aurora is a unified training–serving system that eliminates the "wait time" for speculative decoding by training draft models (speculators) online using live inference traces. By reframing speculative training as an asynchronous reinforcement learning problem, Aurora enables "day-0" deployment of new LLMs with immediate speedups that adapt in real-time to changing user traffic.

What Problem Does This Paper Address?

Speculative decoding is a popular technique to speed up Large Language Model (LLM) inference. It uses a small, fast "draft" model to guess tokens, which a large "target" model then verifies. If the draft is good, you get multiple tokens for the price of one target model step.

However, the authors identify a major "decoupling" problem in how this is currently done:

  1. High Time-to-Serve: You usually have to train the draft model offline for weeks before you can deploy it with a new LLM.
  2. Delayed Utility Feedback: You don't actually know if your draft model will make things faster in production (where system overheads and hardware quirks matter) until it's already deployed.
  3. Domain Drift: If your users suddenly start asking coding questions instead of creative writing, a static draft model becomes "stale" and loses its effectiveness.

Methods

The researchers introduce Aurora, a system that closes the loop between training and serving.

The RL Framing

Aurora treats speculative decoding as an asynchronous Reinforcement Learning (RL) problem.

  • The Policy: The draft model (speculator).
  • The Environment: The target model (verifier).
  • The Reward: Accepted tokens provide positive feedback, while rejected proposals provide implicit negative feedback.

System Architecture

Aurora splits the work into two decoupled components:

  • Inference Server: Based on SGLang, it handles live requests. It streams "traces" (both accepted and rejected tokens) into a data buffer.
  • Training Server: It pulls from that buffer and updates the draft model asynchronously. Once a better version is ready, it "hot-swaps" the weights back into the inference server without any downtime.

Technical Innovations

  • Tree Attention: Since speculative decoding produces branches of possible tokens, Aurora uses a specialized attention mask to train on the entire "tree" of results (both right and wrong) in a single pass.
  • Discard Sampling: A new loss function that explicitly teaches the model not to suggest tokens that were previously rejected.
  • Zero-Copy Target Model: To save memory, the training server doesn't even load the massive target model. It only needs the hidden states and logits sent over from the inference server via a high-speed RPC (Remote Procedure Call) channel.

Datasets

The authors evaluated Aurora using a curated stream of 44,000 prompts across five diverse domains to simulate realistic "distribution shifts":

  • GSM8K: Mathematical reasoning.
  • Spider: Text-to-SQL generation.
  • CodeSearchNet: Programming tasks.
  • Finance-Alpaca: Financial instructions.
  • Chatbot Instruction Prompts: General conversational data.

Key Insights and Ideas

  • The "Day-0" Breakthrough: You don't need a pre-trained draft model. Aurora can start with a completely untrained (randomly initialized) speculator and reach competitive performance within just a few thousand requests.
  • Rejections are Data: Most systems ignore rejected tokens. Aurora proves that "negative feedback" from rejections is a powerful signal for improving sample efficiency.
  • System-Aware Optimization: Because Aurora trains on live production traces, it naturally accounts for system-level overheads (like kernel latency or batching behavior) that offline training ignores.
  • Lazy Synchronization is Better: Surprisingly, updating the model too frequently can hurt performance due to synchronization overhead. A "moderately lazy" schedule (e.g., every 80 requests) provides the best balance of adaptation speed and serving stability.

Key Results

  • Frontier Model Speedups: Aurora achieved a 1.5× speedup on "day-0" for massive models like MiniMax M2.1 (229B) and Qwen3-Coder-Next (80B).
  • Adaptability: When user traffic shifted domains, Aurora delivered an additional 1.25× speedup compared to a well-trained but static (non-adaptive) speculator.
  • Efficiency: The "zero-copy" design allows training a speculator for a 70B model using only the memory required for the small draft model (~1B parameters), making it highly cost-effective.

Limitations and Future Work

  • Batch Size Scaling: At very large batch sizes (e.g., 32+), the overhead of verification can start to eat into the gains of speculative decoding. The authors suggest that "multistep kernels" might be needed to maintain speedups at high concurrency.
  • Hybrid Architectures: While tested on Mixture-of-Experts (MoE) and standard Transformers, the authors note that hybrid models (like those using Mamba blocks) present unique verification overheads that require further optimization.
  • Compute Costs: While memory-efficient, Aurora still requires dedicated GPU compute for the training server, which might be a hurdle for smaller-scale deployments.

Summary updated

Abstract

Speculative decoding can significantly accelerate LLM serving, yet most deployments today disentangle speculator training from serving, treating speculator training as a standalone offline modeling problem. We show that this decoupled formulation introduces substantial deployment and adaptation lag: (1) high time-to-serve, since a speculator must be trained offline for a considerable period before deployment; (2) delayed utility feedback, since the true end-to-end decoding speedup is only known after training and cannot be inferred reliably from acceptance rate alone due to model-architecture and system-level overheads; and (3) domain-drift degradation, as the target model is repurposed to new domains and the speculator becomes stale and less effective. To address these issues, we present Aurora, a unified training-serving system that closes the loop by continuously learning a speculator directly from live inference traces. Aurora reframes online speculator learning as an asynchronous reinforcement-learning problem: accepted tokens provide positive feedback, while rejected speculator proposals provide implicit negative feedback that we exploit to improve sample efficiency. Our design integrates an SGLang-based inference server with an asynchronous training server, enabling hot-swapped speculator updates without service interruption. Crucially, Aurora supports day-0 deployment: a speculator can be served immediately and rapidly adapted to live traffic, improving system performance while providing immediate utility feedback. Across experiments, Aurora achieves a 1.5x day-0 speedup on recently released frontier models (e.g., MiniMax M2.1 229B and Qwen3-Coder-Next 80B). Aurora also adapts effectively to distribution shifts in user traffic, delivering an additional 1.25x speedup over a well-trained but static speculator on widely used models (e.g., Qwen3 and Llama3).
BibTeX
@inproceedings{wang2026when,
  author = {Junxiong Wang and Fengxiang Bie and Jisen Li and Zhongzhu Zhou and Zelei Shao and Yubo Wang and Yinghui Liu and Qingyang Wu and Avner May and Sri Yanamandra and Yineng Zhang and Ce Zhang and Tri Dao and Percy Liang and Ben Athiwaratkun and Shuaiwen Leon Song and Chen Tang and Xiaoxia Wu},
  title = {{When RL Meets Adaptive Speculative Training: A Unified Training-Serving System}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=1U0lhpW0VL},
}
Figure