TL;DR
Decode Early, Verify Late (DEVL) is a training-free framework that allows high-accuracy "offline" speech recognition models to function as ultra-fast streaming recognizers. By treating partial transcripts generated during speech as drafts and verifying them in a single pass at the end of a turn, DEVL achieves near-perfect accuracy parity with offline models while slashing the time-to-final transcript by up to 7.3×.
What Problem Does This Paper Address?
In the world of voice AI, there is a persistent trade-off between accuracy and latency.
The most accurate Automatic Speech Recognition (ASR) models are typically "offline"—they need to process the entire audio clip at once to reach peak performance. However, voice agents (like Siri or AI customer service) need to respond instantly. Waiting for a user to finish speaking and then starting a full transcription process creates a noticeable delay, measured as Time to Final Segment (TTFS).
Existing streaming models solve the speed problem by processing audio in chunks, but they consistently lag behind offline models in accuracy (Word Error Rate, or WER). The authors of this paper ask: Can we get the accuracy of an offline model with the speed of a streaming one without retraining the model?
Methods
The authors introduce DEVL, which applies the principles of speculative decoding to speech recognition. The process works in two main phases:
- Decode Early (The "Draft"): While the user is still speaking, the server periodically generates "checkpoints." It encodes the audio received so far and generates a partial transcript, which it stores as a draft. Because it uses a paged KV (Key-Value) cache, it can reuse previous computations as more audio arrives.
- Verify Late (The "Commit"): When the user stops speaking (the "commit"), the server doesn't start from scratch. Instead, it takes the latest draft and runs it through the model alongside the complete audio in a single forward pass.
- If the draft matches the model's logic for the full audio, it is accepted.
- If there is a mismatch, the system "rolls back" to the last correct word and finishes the transcription from that point.
To ensure the final output is identical to a standard offline decode, the authors implemented specific design choices, such as encoding audio in fixed blocks and maintaining deterministic front-end states. This ensures that the "streaming" environment perfectly mimics the "offline" environment the model was designed for.
Datasets
The researchers evaluated DEVL using two primary benchmarks:
- Pipecat-STT: A dataset consisting of 897 voice-agent turns. The reference transcripts were generated by Gemini.
- WildASR: A more challenging set of 2,549 English clips featuring diverse recording conditions and accented speech.
They tested these datasets across five different model families, including Qwen3-ASR, Whisper (large-v3), VibeVoice-ASR, and NVIDIA-Nemotron.
Key Insights and Ideas
- The "Draft" is surprisingly good: The authors found that for Qwen3-ASR, the last partial transcript generated during speech typically matches the first 82–88% of the final transcript. This high "hit rate" is why verification is so much faster than re-decoding.
- Training-Free Versatility: Unlike other streaming methods that require specialized models or fine-tuning, DEVL works with off-the-shelf weights. The authors successfully adapted it to four distinct model architectures, including Transformers and Mamba-2 based models.
- Exact Arithmetic Parity: The paper provides a "correctness sketch" proving that, under specific conditions, DEVL’s output is mathematically identical to the model’s greedy offline output. You aren't just getting "close" to offline accuracy; you are getting the exact same result faster.
- Smart Scheduling: To handle high traffic, the DEVL scheduler prioritizes "finals" (the last step to give the user an answer) over "drafts" (the background work). This ensures that even when the GPU is busy, the user-facing latency stays low.
Key Results
- Massive Speedups: Single-stream time-to-final (TTFS) was reduced by 1.5–7.3× compared to standard offline decoding.
- Ultra-Low Latency: When serving Qwen3-ASR-1.7B on an NVIDIA A100, DEVL returned final transcripts in a median of 22–26 ms after the user stopped speaking.
- Accuracy Preservation: Between 95% and 100% of final transcripts were byte-identical to the offline versions. The Word Error Rate (WER) stayed within a negligible 0.13 points of the offline baseline.
- High Throughput: DEVL was the only system tested that could meet a 100 ms "p99" latency target while serving up to 80 concurrent streams on a single GPU.
Limitations and Future Work
- English Only: The benchmarks were limited to English audio. Performance on tonal languages or highly inflected languages remains to be seen.
- Greedy Decoding Only: The current framework requires greedy decoding. While greedy decoding is standard for many ASR tasks, DEVL does not yet support beam search.
- Partial Latency: While DEVL makes the final transcript arrive much faster, it doesn't necessarily improve the latency of the "scrolling" text the user sees while they are still talking.
- GPU Non-determinism: The authors noted that batch-dependent GPU rounding can occasionally cause tiny variations in results, though this could potentially be solved with batch-invariant kernels.
Summary updated