TL;DR
Aurora is a unified training–serving system that eliminates the "wait time" for speculative decoding by training draft models (speculators) online using live inference traces. By reframing speculative training as an asynchronous reinforcement learning problem, Aurora enables "day-0" deployment of new LLMs with immediate speedups that adapt in real-time to changing user traffic.
What Problem Does This Paper Address?
Speculative decoding is a popular technique to speed up Large Language Model (LLM) inference. It uses a small, fast "draft" model to guess tokens, which a large "target" model then verifies. If the draft is good, you get multiple tokens for the price of one target model step.
However, the authors identify a major "decoupling" problem in how this is currently done:
- High Time-to-Serve: You usually have to train the draft model offline for weeks before you can deploy it with a new LLM.
- Delayed Utility Feedback: You don't actually know if your draft model will make things faster in production (where system overheads and hardware quirks matter) until it's already deployed.
- Domain Drift: If your users suddenly start asking coding questions instead of creative writing, a static draft model becomes "stale" and loses its effectiveness.
Methods
The researchers introduce Aurora, a system that closes the loop between training and serving.
The RL Framing
Aurora treats speculative decoding as an asynchronous Reinforcement Learning (RL) problem.
- The Policy: The draft model (speculator).
- The Environment: The target model (verifier).
- The Reward: Accepted tokens provide positive feedback, while rejected proposals provide implicit negative feedback.
System Architecture
Aurora splits the work into two decoupled components:
- Inference Server: Based on SGLang, it handles live requests. It streams "traces" (both accepted and rejected tokens) into a data buffer.
- Training Server: It pulls from that buffer and updates the draft model asynchronously. Once a better version is ready, it "hot-swaps" the weights back into the inference server without any downtime.
Technical Innovations
- Tree Attention: Since speculative decoding produces branches of possible tokens, Aurora uses a specialized attention mask to train on the entire "tree" of results (both right and wrong) in a single pass.
- Discard Sampling: A new loss function that explicitly teaches the model not to suggest tokens that were previously rejected.
- Zero-Copy Target Model: To save memory, the training server doesn't even load the massive target model. It only needs the hidden states and logits sent over from the inference server via a high-speed RPC (Remote Procedure Call) channel.
Datasets
The authors evaluated Aurora using a curated stream of 44,000 prompts across five diverse domains to simulate realistic "distribution shifts":
- GSM8K: Mathematical reasoning.
- Spider: Text-to-SQL generation.
- CodeSearchNet: Programming tasks.
- Finance-Alpaca: Financial instructions.
- Chatbot Instruction Prompts: General conversational data.
Key Insights and Ideas
- The "Day-0" Breakthrough: You don't need a pre-trained draft model. Aurora can start with a completely untrained (randomly initialized) speculator and reach competitive performance within just a few thousand requests.
- Rejections are Data: Most systems ignore rejected tokens. Aurora proves that "negative feedback" from rejections is a powerful signal for improving sample efficiency.
- System-Aware Optimization: Because Aurora trains on live production traces, it naturally accounts for system-level overheads (like kernel latency or batching behavior) that offline training ignores.
- Lazy Synchronization is Better: Surprisingly, updating the model too frequently can hurt performance due to synchronization overhead. A "moderately lazy" schedule (e.g., every 80 requests) provides the best balance of adaptation speed and serving stability.
Key Results
- Frontier Model Speedups: Aurora achieved a 1.5× speedup on "day-0" for massive models like MiniMax M2.1 (229B) and Qwen3-Coder-Next (80B).
- Adaptability: When user traffic shifted domains, Aurora delivered an additional 1.25× speedup compared to a well-trained but static (non-adaptive) speculator.
- Efficiency: The "zero-copy" design allows training a speculator for a 70B model using only the memory required for the small draft model (~1B parameters), making it highly cost-effective.
Limitations and Future Work
- Batch Size Scaling: At very large batch sizes (e.g., 32+), the overhead of verification can start to eat into the gains of speculative decoding. The authors suggest that "multistep kernels" might be needed to maintain speedups at high concurrency.
- Hybrid Architectures: While tested on Mixture-of-Experts (MoE) and standard Transformers, the authors note that hybrid models (like those using Mamba blocks) present unique verification overheads that require further optimization.
- Compute Costs: While memory-efficient, Aurora still requires dedicated GPU compute for the training server, which might be a hurdle for smaller-scale deployments.
Summary updated