Abstract
Speech large language models (LLMs) have emerged as a prominent research focus in speech processing. In this work, we introduce VocalNet, a series of high-performance speech LLMs featuring a scalable and model-agnostic training framework as well as a novel multi-token prediction (MTP) paradigm for speech generation. We first propose an efficient two-stage training framework that enables LLMs to acquire real-time speech interaction capabilities. Furthermore, we introduce MTP into the domain of speech LLMs—an alternative to traditional next-token prediction (NTP)—which enables the model to predict multiple future tokens at each step. Experimental results demonstrate that VocalNet achieves performance comparable to state-of-the-art Omni LLMs while outperforming existing open-source speech LLMs, despite using limited training data.
Introduction
Modern speech interaction systems are transitioning from traditional cascaded pipelines (ASR → LLM → TTS) toward end-to-end models to minimize latency and information loss. However, these end-to-end models often struggle with complex, multi-stage training recipes and the inherent limitations of Next-Token Prediction (NTP). Because acoustic units typically span multiple tokens, sequential one-by-one generation is slow and often fails to capture the hierarchical structure of speech. VocalNet addresses these challenges by introducing a streamlined training framework and a novel Multi-Token Prediction (MTP) paradigm.
Architecture and Two-Stage Training
VocalNet utilizes an aligned multimodal architecture that preserves the reasoning power of a text-based LLM (such as Llama-3) while integrating speech-specific components. The system includes a Whisper-large-v3 encoder for input processing, a custom speech decoder, and a flow-matching vocoder based on CosyVoice2 for high-quality audio synthesis.
To ensure scalability and simplicity, VocalNet employs a two-stage training process:
- Multi-modal Alignment: The model is trained to bridge speech and text modalities, enabling speech-to-text understanding.
- Generative Fine-Tuning: The model is fine-tuned to generate high-quality speech responses directly, enabling seamless speech-to-speech interaction.
The Multi-Token Prediction (MTP) Paradigm
The core innovation of VocalNet is the introduction of Multi-Token Prediction. Unlike traditional NTP, which predicts only the immediate next token, VocalNet uses sequential Transformer layers as MTP modules to predict $N$ future tokens simultaneously.
Crucially, VocalNet's MTP implementation utilizes previously computed hidden states rather than ground-truth tokens during training. This approach preserves causal dependencies and allows the model to better capture local speech structures, such as the flow of a syllable. This paradigm shift does not just accelerate inference; it fundamentally improves speech quality by reducing error accumulation (drift) during long audio generations.
Experimental Results
VocalNet was evaluated on the OpenAudioBench suite using a relatively modest training set of ~6,000 hours of speech. Despite this limited data, the model achieved state-of-the-art results across several benchmarks.
As shown in the results, VocalNet-8B achieved a UTMOS score of 4.49, tying for the highest naturalness score among all tested models. Furthermore, the MTP paradigm contributed to a significant reduction in Word Error Rate (WER), with VocalNet-1B outperforming larger competitors. In terms of responsiveness, VocalNet-1B achieves an end-to-end latency of approximately 320ms, while the 8B variant sits at 430ms, making it highly suitable for real-time applications.
Conclusion
VocalNet demonstrates that high-quality speech generation in LLMs can be achieved through architectural simplicity and smarter prediction paradigms. By moving beyond sequential next-token prediction to a multi-token approach, VocalNet achieves a rare synergy where increased generation speed is accompanied by superior audio quality. Future work will focus on improving paralinguistic controllability and further optimizing the vocoder to reduce the remaining latency bottlenecks.
Summary updated