Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR

Yael Segal-Feldman, Aviv Shamsian, Aviv Navon, Gill Hetz, Joseph Keshet

TL;DR

Whisper-Medusa speeds up OpenAI’s Whisper ASR model by predicting multiple tokens simultaneously rather than one by one. By adding "Medusa heads" to the existing architecture, the researchers achieved up to a 50% reduction in latency with minimal impact on transcription accuracy, effectively making one of the world's best speech models significantly faster.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) has been revolutionized by Transformer-based models like OpenAI’s Whisper. However, these models are "autoregressive," meaning they generate text one token at a time. For a large model like Whisper (1.5 billion parameters), this sequential process is computationally expensive and slow, leading to high latency in real-world applications.

While techniques like "speculative decoding" exist to speed up inference, they typically require a second, smaller "assistant" model to draft candidate tokens. Maintaining two separate models is a headache for engineers. This paper tackles the challenge of self-assisted speculative decoding: how can we make Whisper faster by modifying its own architecture to predict multiple future tokens at once, without needing a separate draft model?

Methods

The authors introduce Whisper-Medusa, which adds $K$ additional "heads" to the Whisper decoder. While the standard "base head" predicts the next token ($y_t$), the Medusa heads predict subsequent tokens ($y_{t+1}, y_{t+2}, \dots, y_{t+k}$) in parallel.

They proposed two specific architectural variations:

  1. Medusa-Linear: Each extra head is a simple linear layer with a residual connection. This is lightweight and adds only about 18M parameters.
  2. Medusa-Block: This version adds an entire Transformer decoder block shared across the heads before the final linear layers. This is more robust but adds about 42M parameters.

The Inference Logic: Inference happens in two stages. First, the model predicts a block of $K+1$ tokens. Second, the "base head" verifies these candidates. Instead of a simple "yes/no," they use a sophisticated thresholding method based on the entropy of the probability distribution. If the base head confirms the candidates are likely enough, they are accepted, skipping several rounds of standard computation.

Datasets

The researchers evaluated their approach across several benchmarks:

  • LibriSpeech: The gold standard for English speech recognition (approx. 1,000 hours).
  • VoxPopuli: A large-scale multilingual corpus from European Parliament recordings. They tested on English, German, Spanish, and French.
  • Self-Supervised VoxPopuli: They also experimented with 400,000 hours of unlabeled audio for Czech, Finnish, and Dutch to see how the model performs when trained on "pseudo-labels" generated by Whisper itself.

Key Insights and Ideas

  • ASR is uniquely suited for Medusa: Unlike standard Large Language Models (LLMs) which are decoder-only, ASR models have an encoder that processes the entire audio chunk upfront. This provides a rich context that makes predicting multiple future tokens more stable than in pure text generation.
  • The "Sweet Spot" for Heads: Through ablation studies, the authors found that 10 Medusa heads provided the best balance. Adding more (15 or 20) actually caused speed and accuracy to decline, likely because the task of predicting 20 tokens ahead is too difficult for the model to learn accurately.
  • Minimal Accuracy Trade-off: Surprisingly, the Medusa-Block architecture often stayed within a fraction of a percentage point of the original Whisper’s Word Error Rate (WER), and in some cases (like LibriSpeech Test-Clean), it actually showed slight improvements.

Key Results

  • Latency Reduction: The models achieved speedups ranging from 1.23x to 1.83x. In practical terms, this is a 20% to 50% reduction in the time it takes to transcribe audio.
  • Efficiency: Medusa-Linear is the "speed demon," offering the highest latency reductions, while Medusa-Block is the "accuracy specialist," maintaining the highest transcription quality.
  • Multilingual Success: The approach worked well across different languages, though performance varied based on how much training data was available for each specific language.

Limitations and Future Work

The authors noted a few hurdles:

  • Data Imbalance: The Medusa-Linear model is sensitive to imbalanced datasets. If a language has very little data, the extra heads struggle to learn the complex patterns needed for multi-token prediction.
  • Greedy Decoding: Currently, the system uses "greedy decoding" (picking the single most likely token). The authors want to extend this to Beam Search, which explores multiple possible paths and usually results in even higher accuracy.
  • Open Source: To encourage further optimization, the team has released their code on GitHub, allowing the community to apply Medusa heads to other Transformer-based speech models.

Summary updated

Abstract

Large transformer-based models have significant potential for speech transcription and translation. Their self-attention mechanisms and parallel processing enable them to capture complex patterns and dependencies in audio sequences. However, this potential comes with challenges, as these large and computationally intensive models lead to slow inference speeds. Various optimization strategies have been proposed to improve performance, including efficient hardware utilization and algorithmic enhancements. In this paper, we introduce Whisper-Medusa, a novel approach designed to enhance processing speed with minimal impact on Word Error Rate (WER). The proposed model extends the OpenAI’s Whisper architecture by predicting multiple tokens per iteration, resulting in a 50% reduction in latency. We showcase the effectiveness of Whisper-Medusa across different learning setups and datasets.
BibTeX
@inproceedings{segalfeldman2025whisper,
  author = {Yael Segal-Feldman and Aviv Shamsian and Aviv Navon and Gill Hetz and Joseph Keshet},
  title = {{Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=rLxPLOhrxZ},
}

Loading related content...

Figure