TL;DR
Instead of waiting for a user to finish typing a prompt, "Streaming Generation" begins drafting and refining responses sentence-by-sentence during the composition process. By utilizing the idle time while a user types, this approach can reduce the post-submission wait time (residual latency) by over 4x for coding tasks while maintaining high output quality.
What Problem Does This Paper Address?
In standard LLM deployments, the "clock starts" only after a user hits submit. However, composing a complex prompt—like a coding specification or a detailed edit—takes several seconds. During this interval, the model sits idle, even though the request is already partially specified.
The authors argue that the metric we should care about isn't total decoding time, but residual cost: the computation and time required after the final token of the request is available. Current efficiency research focuses on making decoding faster (e.g., speculative decoding), but this paper explores how to hide that decoding time entirely by overlapping it with the user's typing.
Methods
The authors propose a draft–verify–regenerate loop that operates as the user types. They model the input as a sequence of arriving sentences ($S_1, S_2, \dots, S_n$).
- Verify: When a new sentence $S_i$ arrives, the model performs one forward pass over the new input concatenated with the previous draft ($D_{i-1}$). It calculates the probability of each token in the existing draft given the updated context.
- Accept: The system keeps the longest leading run of draft tokens that exceed a probability threshold $\tau$.
- Regenerate: The model resumes decoding from the end of that accepted prefix to produce a new, more complete draft ($D_i$).
A crucial distinction is made between Autoregressive (AR) models and Diffusion models. AR models are "irreversible"—once a token is accepted, it's fixed. Diffusion models (specifically "slot-based" ones like ReFusion) can re-predict every position in their buffer, allowing them to "change their mind" about earlier parts of the answer as more of the prompt arrives.
Datasets
The researchers evaluated their method across six diverse benchmarks:
- Program Synthesis: HumanEval and MBPP.
- Data Science Code Completion: DS-1000.
- Code/Text Editing: CanItEdit and CoEdIT.
- Dialogue: Multi-turn chat (UltraChat-style).
For model adaptation (fine-tuning models to handle incomplete prompts), they used the MBPP training split, Magicoder-OSS-Instruct, and Evol-CodeAlpaca for coding, and LMSYS-Chat-1M for general dialogue.
Key Insights and Ideas
- The "Causal" Requirement: The method fails on fully bidirectional diffusion models (like LLaDA). Because these models see the "future" tokens in a draft during verification, they tend to approve almost anything—even a draft answering a completely different question. Causal attention is required to make verification meaningful.
- Task Dependency: Streaming works best for "specification-first" tasks. In coding (HumanEval/MBPP), the first sentence often defines the entire logic, allowing the model to draft the correct solution early.
- The Quality-Speed Trade-off: For AR models, streaming often incurs a quality penalty because the model commits to an opening before seeing the full prompt. Diffusion models avoid this because they can revise the "accepted" prefix during the final generation step.
- Asymmetric Risk: A rejected draft only costs one extra verification pass (a bounded loss), while an accepted draft can save hundreds of decoding steps (an unbounded gain). This makes an "unconditional" streaming policy viable.
Key Results
The gains are most dramatic in coding tasks where the prompt's intent is clear early on.
- ReFusion-8B (Diffusion): On HumanEval, the model achieved a 4.67x speedup in forward passes and a 3.68x speedup in wall-clock time, with zero loss in quality.
- Gemma-3-12B (AR): Showed an incredible 14.4x median residual speedup, though it suffered a quality drop on coding benchmarks.
- Distribution of Gain: The benefits are highly right-skewed. As seen in the HumanEval results (Figure 1), most requests see a 2x-8x speedup, but some reach over 20x. Conversely, in open-ended Chat (Figure 6), the median speedup is much lower (1.20x) because semantically valid drafts often fail strict token-level verification.

Limitations and Future Work
- Compute vs. Latency: This is not a "free lunch." Streaming increases the total number of forward passes (by 1.3x to 5.4x) because the model regenerates answers repeatedly. It trades throughput for individual user latency, making it ideal for local or underutilized deployments rather than high-traffic batch systems.
- Irreversibility: The quality drop in AR models remains a hurdle. Future work might explore "soft" commitments or better adaptation techniques to help AR models handle the uncertainty of incomplete prompts.
- Granularity: The study used sentence-level updates as a proxy for typing. Real-world performance may vary based on actual keystroke-level latency and network overhead.
Summary updated