Introduction
Researchers from Tsinghua University and Zhipu.AI have introduced a novel scaling strategy for Speech Language Models (SpeechLMs) by leveraging massive amounts of synthetic data. By generating 600 billion tokens of synthetic interleaved speech-text data, they circumvented the scarcity of parallel speech-text datasets. Training a 9B parameter model on a total of 1 trillion tokens resulted in a significant leap in Spoken QA performance, jumping from a previous state-of-the-art (SOTA) of 13% to 31%.
While Large Language Models (LLMs) benefit from trillions of tokens available in web-scale text corpora, SpeechLMs face a severe data bottleneck. Unsupervised speech data is roughly 1,000x scarcer than text, and high-quality parallel speech-text data is even rarer. Traditional cascaded systems (ASR → LLM → TTS) often suffer from information loss and high latency. End-to-end SpeechLMs offer a more integrated solution, but their development has been hindered by the lack of diverse, large-scale datasets necessary to align vocal nuances with complex reasoning.
Methodology: Synthetic Interleaved Data Pipeline
The core innovation lies in a pipeline that transforms vast text corpora into "interleaved" speech-text data without the need for intermediate audio file generation during pre-training. This approach utilizes a supervised speech tokenizer and a high-speed "synthetic data factory."
Two-Stage Training Framework
The training process is divided into two distinct stages:
- Stage I (Pre-training): A 9B parameter base model (GLM-4-9B) was trained on 1 trillion tokens. This included 600B synthetic interleaved tokens created via "span corruption," where random text segments in high-quality corpora (FineWeb-Edu) were replaced with synthetic speech tokens generated by a 1.5B parameter Text-to-Token model.
- Stage II (Fine-tuning): The model was refined using SpeechDialog-90K, a curated dataset of dialogues adapted by GPT-4 into speech-friendly formats to enable natural spoken chatbot interactions.
Key Results and Scaling Laws
The experimental results demonstrate that interleaving is crucial for cross-modal alignment. By mixing modalities at the word level, the model learns to map text-based knowledge directly into the speech domain.
Spoken QA Breakthrough
On the Llama Questions benchmark, the 9B model achieved 50.7% accuracy in speech-to-speech mode, more than doubling the performance of the previous SOTA, Moshi (21.0%). In "text-guided" mode, accuracy reached 64.7%, indicating that the model effectively transfers reasoning capabilities from text to speech.
The Impact of Scaling
The study confirms a clear scaling law: performance on spoken tasks improves consistently as the volume of synthetic interleaved data increases. The model successfully outperformed previous state-of-the-art models across nearly all metrics despite having access to less "natural" speech data.
Discussion and Limitations
A key finding was that lower frame rates (12.5Hz vs. 50Hz) are essential for scaling, as they mitigate the "token expansion" problem and allow the model to process more data within a fixed computational budget. However, the authors noted that synthetic data currently has a higher Word Error Rate (WER) of 9-11% compared to 3% for standard TTS datasets. This is attributed to the difficulty the Text-to-Token model faces with rare or complex words found in diverse web text. Future work will focus on improving the prosody and emotional nuance of synthetic tokens and expanding the pipeline to support multilingual scaling.
Summary updated