VocalNet: Speech LLMs with Multi-Token Prediction for Faster and High-Quality Generation

Yuhao Wang, Heyang Liu, Ziyang Cheng, Ronghua Wu, Qunshan Gu, Yanfeng WangEMNLP 2025 · OpenReview

Abstract

Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. In this work, we introduce VocalNet, a series of high-performance speech LLMs featuring a scalable and model-agnostic training framework as well as a novel multi-token prediction (MTP) paradigm for speech generation. We first propose an efficient two-stage training framework that enables LLMs to acquire real-time speech interaction capabilities. Furthermore, we introduce MTP into the domain of speech LLMs—an alternative to traditional next-token prediction (NTP)—which enables the model to predict multiple future tokens at each step. Experimental results demonstrate that VocalNet achieves performance comparable to state-of-the-art Omni LLMs while outperforming existing open-source speech LLMs, despite using limited training data.

Introduction

Modern speech interaction systems are transitioning from traditional cascaded pipelines (ASR → LLM → TTS) toward end-to-end models to minimize latency and information loss. However, these end-to-end models often struggle with complex, multi-stage training recipes and the inherent limitations of Next-Token Prediction (NTP). Because acoustic units typically span multiple tokens, sequential one-by-one generation is slow and often fails to capture the hierarchical structure of speech. VocalNet addresses these challenges by introducing a streamlined training framework and a novel Multi-Token Prediction (MTP) paradigm.

Architecture and Two-Stage Training

VocalNet utilizes an aligned multimodal architecture that preserves the reasoning power of a text-based LLM (such as Llama-3) while integrating speech-specific components. The system includes a Whisper-large-v3 encoder for input processing, a custom speech decoder, and a flow-matching vocoder based on CosyVoice2 for high-quality audio synthesis.

Overview of the VocalNet architecture and its two-stage training framework: Multi-modal Alignment and Generative Supervised Fine-Tuning.

To ensure scalability and simplicity, VocalNet employs a two-stage training process:

  1. Multi-modal Alignment: The model is trained to bridge speech and text modalities, enabling speech-to-text understanding.
  2. Generative Fine-Tuning: The model is fine-tuned to generate high-quality speech responses directly, enabling seamless speech-to-speech interaction.

The Multi-Token Prediction (MTP) Paradigm

The core innovation of VocalNet is the introduction of Multi-Token Prediction. Unlike traditional NTP, which predicts only the immediate next token, VocalNet uses sequential Transformer layers as MTP modules to predict $N$ future tokens simultaneously.

Crucially, VocalNet's MTP implementation utilizes previously computed hidden states rather than ground-truth tokens during training. This approach preserves causal dependencies and allows the model to better capture local speech structures, such as the flow of a syllable. This paradigm shift does not just accelerate inference; it fundamentally improves speech quality by reducing error accumulation (drift) during long audio generations.

Experimental Results

VocalNet was evaluated on the OpenAudioBench suite using a relatively modest training set of ~6,000 hours of speech. Despite this limited data, the model achieved state-of-the-art results across several benchmarks.

Comparison of Word Error Rate (WER) and UTMOS (speech naturalness) across various open-source speech LLMs.

As shown in the results, VocalNet-8B achieved a UTMOS score of 4.49, tying for the highest naturalness score among all tested models. Furthermore, the MTP paradigm contributed to a significant reduction in Word Error Rate (WER), with VocalNet-1B outperforming larger competitors. In terms of responsiveness, VocalNet-1B achieves an end-to-end latency of approximately 320ms, while the 8B variant sits at 430ms, making it highly suitable for real-time applications.

Conclusion

VocalNet demonstrates that high-quality speech generation in LLMs can be achieved through architectural simplicity and smarter prediction paradigms. By moving beyond sequential next-token prediction to a multi-token approach, VocalNet achieves a rare synergy where increased generation speed is accompanied by superior audio quality. Future work will focus on improving paralinguistic controllability and further optimizing the vocoder to reduce the remaining latency bottlenecks.

Summary updated

Abstract

Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. In this work, we introduce VocalNet, a series of high-performance speech LLMs featuring a scalable and model-agnostic training framework as well as a novel multi-token prediction (MTP) paradigm for speech generation. We first propose an efficient two-stage training framework that enables LLMs to acquire real-time speech interaction capabilities. Through extensive experiments on various training configurations, we ensure both simplicity and effectiveness in the training strategy. Furthermore, inspired by advances in language modeling, we introduce MTP into the domain of speech LLMs—an alternative to traditional next-token prediction (NTP)—which enables the model to predict multiple future tokens at each step. Through systematic analysis and improved implementation, we show that MTP not only accelerates inference speed but also significantly enhances speech quality. Experimental results demonstrate that VocalNet achieves performance comparable to state-of-the-art Omni LLMs while outperforming existing open-source speech LLMs, despite using limited training data.
BibTeX
@inproceedings{wang2025vocalnet,
  author = {Yuhao Wang and Heyang Liu and Ziyang Cheng and Ronghua Wu and Qunshan Gu and Yanfeng Wang and Yu Wang},
  title = {{VocalNet: Speech LLMs with Multi-Token Prediction for Faster and High-Quality Generation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  url = {https://openreview.net/forum?id=0VvTdOG0mp},
}
Figure