Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR

Yael Segal-Feldman, Aviv Shamsian, Aviv Navon, Gill Hetz, Joseph Keshet

Abstract

Large transformer-based models have significant potential for speech transcription and translation. Their self-attention mechanisms and parallel processing enable them to capture complex patterns and dependencies in audio sequences. However, this potential comes with challenges, as these large and computationally intensive models lead to slow inference speeds. Various optimization strategies have been proposed to improve performance, including efficient hardware utilization and algorithmic enhancements. In this paper, we introduce Whisper-Medusa, a novel approach designed to enhance processing speed with minimal impact on Word Error Rate (WER). The proposed model extends the OpenAI's Whisper architecture by predicting multiple tokens per iteration, resulting in a 50% reduction in latency. We showcase the effectiveness of Whisper-Medusa across different learning setups and datasets.

Introduction

Automatic Speech Recognition (ASR) has been revolutionized by Transformer-based models like OpenAI’s Whisper. However, these models are typically "autoregressive," generating text one token at a time. For large-scale models like Whisper (1.5 billion parameters), this sequential process is computationally expensive and slow, leading to high latency in real-world applications.

While techniques like speculative decoding exist to speed up inference, they often require a second, smaller "assistant" model to draft candidate tokens, which increases engineering overhead. This paper introduces Whisper-Medusa, a self-assisted speculative decoding approach that modifies Whisper's own architecture to predict multiple future tokens simultaneously, eliminating the need for a separate draft model.

Methods

Whisper-Medusa adds $K$ additional "heads" to the Whisper decoder. While the standard "base head" predicts the next token ($y_t$), the Medusa heads predict subsequent tokens ($y_{t+1}, y_{t+2}, \dots, y_{t+k}$) in parallel. The authors proposed two specific architectural variations:

  1. Medusa-Linear: Each extra head is a simple linear layer with a residual connection, adding approximately 18M parameters.
  2. Medusa-Block: This version adds an entire Transformer decoder block shared across the heads before the final linear layers, adding about 42M parameters.
Architectural diagrams of Medusa-Linear (left) and Medusa-Block (right) integrated into the Whisper framework.

Inference occurs in two stages: first, the model predicts a block of $K+1$ tokens; second, the base head verifies these candidates using an entropy-based thresholding method. If the candidates are deemed likely enough, they are accepted, allowing the model to skip multiple rounds of standard sequential computation.

Key Results and Insights

The researchers evaluated Whisper-Medusa across several benchmarks, including LibriSpeech and VoxPopuli. A key insight is that ASR is uniquely suited for Medusa-style decoding because the encoder processes the entire audio chunk upfront, providing a rich context that stabilizes multi-token prediction.

Performance comparison on LibriSpeech Test Clean and Test Other datasets showing WER, CER, and Speedup.
  • Latency Reduction: The models achieved speedups ranging from 1.23x to 1.83x, representing a 20% to 50% reduction in transcription time.
  • Accuracy vs. Efficiency: Medusa-Linear offers the highest latency reductions, while Medusa-Block maintains the highest transcription quality, often staying within a fraction of a percentage point of the original Whisper’s Word Error Rate (WER).
  • The "Sweet Spot" for Heads: Ablation studies found that 10 Medusa heads provided the best balance. Beyond this, accuracy and speed declined as the task of predicting many tokens ahead became too difficult for the model to learn accurately.

Limitations and Future Work

Despite its success, the Medusa-Linear model showed sensitivity to imbalanced datasets, particularly in low-resource languages. Additionally, the current implementation uses greedy decoding; the authors intend to extend this to beam search to further improve accuracy. To encourage further optimization, the team has released their code on GitHub, allowing the community to apply Medusa heads to other Transformer-based speech models.

Summary updated

Abstract

Large transformer-based models have significant potential for speech transcription and translation. Their self-attention mechanisms and parallel processing enable them to capture complex patterns and dependencies in audio sequences. However, this potential comes with challenges, as these large and computationally intensive models lead to slow inference speeds. Various optimization strategies have been proposed to improve performance, including efficient hardware utilization and algorithmic enhancements. In this paper, we introduce Whisper-Medusa, a novel approach designed to enhance processing speed with minimal impact on Word Error Rate (WER). The proposed model extends the OpenAI's Whisper architecture by predicting multiple tokens per iteration, resulting in a 50% reduction in latency. We showcase the effectiveness of Whisper-Medusa across different learning setups and datasets.
BibTeX
@inproceedings{segalfeldman2025whisper,
  author = {Yael Segal-Feldman and Aviv Shamsian and Aviv Navon and Gill Hetz and Joseph Keshet},
  title = {{Whisper in Medusa's Ear: Multi-head Efficient Decoding for Transformer-based ASR}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=rLxPLOhrxZ},
}
Figure