Scaling Speech-Text Pre-training with Synthetic Interleaved Data

Aohan Zeng, Zhengxiao Du, Mingdao Liu, Lei Zhang, Shengmin Jiang, Yuxiao Dong, Jie Tang

Introduction

Researchers from Tsinghua University and Zhipu.AI have introduced a novel scaling strategy for Speech Language Models (SpeechLMs) by leveraging massive amounts of synthetic data. By generating 600 billion tokens of synthetic interleaved speech-text data, they circumvented the scarcity of parallel speech-text datasets. Training a 9B parameter model on a total of 1 trillion tokens resulted in a significant leap in Spoken QA performance, jumping from a previous state-of-the-art (SOTA) of 13% to 31%.

While Large Language Models (LLMs) benefit from trillions of tokens available in web-scale text corpora, SpeechLMs face a severe data bottleneck. Unsupervised speech data is roughly 1,000x scarcer than text, and high-quality parallel speech-text data is even rarer. Traditional cascaded systems (ASR → LLM → TTS) often suffer from information loss and high latency. End-to-end SpeechLMs offer a more integrated solution, but their development has been hindered by the lack of diverse, large-scale datasets necessary to align vocal nuances with complex reasoning.

Methodology: Synthetic Interleaved Data Pipeline

The core innovation lies in a pipeline that transforms vast text corpora into "interleaved" speech-text data without the need for intermediate audio file generation during pre-training. This approach utilizes a supervised speech tokenizer and a high-speed "synthetic data factory."

Two-Stage Training Framework

The training process is divided into two distinct stages:

  1. Stage I (Pre-training): A 9B parameter base model (GLM-4-9B) was trained on 1 trillion tokens. This included 600B synthetic interleaved tokens created via "span corruption," where random text segments in high-quality corpora (FineWeb-Edu) were replaced with synthetic speech tokens generated by a 1.5B parameter Text-to-Token model.
  2. Stage II (Fine-tuning): The model was refined using SpeechDialog-90K, a curated dataset of dialogues adapted by GPT-4 into speech-friendly formats to enable natural spoken chatbot interactions.
Overview of the two-stage training framework, including Text-to-Token model training, interleaved data construction, and the pre-training/fine-tuning stages.

Key Results and Scaling Laws

The experimental results demonstrate that interleaving is crucial for cross-modal alignment. By mixing modalities at the word level, the model learns to map text-based knowledge directly into the speech domain.

Spoken QA Breakthrough

On the Llama Questions benchmark, the 9B model achieved 50.7% accuracy in speech-to-speech mode, more than doubling the performance of the previous SOTA, Moshi (21.0%). In "text-guided" mode, accuracy reached 64.7%, indicating that the model effectively transfers reasoning capabilities from text to speech.

The Impact of Scaling

The study confirms a clear scaling law: performance on spoken tasks improves consistently as the volume of synthetic interleaved data increases. The model successfully outperformed previous state-of-the-art models across nearly all metrics despite having access to less "natural" speech data.

Accuracy on Spoken QA benchmarks scales significantly with the amount of synthetic interleaved data, surpassing previous state-of-the-art models like Moshi.

Discussion and Limitations

A key finding was that lower frame rates (12.5Hz vs. 50Hz) are essential for scaling, as they mitigate the "token expansion" problem and allow the model to process more data within a fixed computational budget. However, the authors noted that synthetic data currently has a higher Word Error Rate (WER) of 9-11% compared to 3% for standard TTS datasets. This is attributed to the difficulty the Text-to-Token model faces with rare or complex words found in diverse web text. Future work will focus on improving the prosody and emotional nuance of synthetic tokens and expanding the pipeline to support multilingual scaling.

Summary updated

BibTeX
@inproceedings{zeng2025scaling,
  author = {Aohan Zeng and Zhengxiao Du and Mingdao Liu and Lei Zhang and Shengmin Jiang and Yuxiao Dong and Jie Tang},
  title = {{Scaling Speech-Text Pre-training with Synthetic Interleaved Data}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=3tukjsVyrE},
}
Figure