Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners

Rongjie Huang, Chunlei Zhang, Yongqi Wang, Dongchao Yang, Jinchuan Tian, Zhenhui Ye, Luping Liu, Zehan Wang, Ziyue Jiang, Xuankai Chang, Jiatong Shi, Chao Weng, Zhou Zhao, Dong Yu

Abstract

Make-A-Voice is a unified, decoder-only voice Large Language Model (LLM) that treats various voice generation tasks—such as text-to-speech and singing synthesis—as a language modeling problem. By scaling to 1.2 billion parameters and training on 200,000 hours of multilingual data, it achieves state-of-the-art performance in zero-shot voice cloning. This research demonstrates that the "scaling hypothesis" holds for audio as it does for text, showing that larger models and diverse datasets significantly enhance audio quality and cross-task generalization.

Introduction

While text-based Large Language Models (LLMs) like GPT-4 have revolutionized natural language processing by serving as general-purpose interfaces, voice-based models have historically remained specialized. Existing "Voice LLMs" often excel in specific domains, such as English text-to-speech (TTS), but struggle with diverse tasks like singing synthesis or low-resource languages.

The authors of Make-A-Voice identify three critical gaps in the current landscape:

  1. Scalability: The impact of scaling model parameters and data volume on voice quality remains under-explored.
  2. Multitask Learning: Most models lack the ability to share knowledge across related audio tasks.
  3. Low-Resource Support: Data scarcity hinders the development of high-quality models for non-dominant languages.

Methods: A Voice-as-Language Approach

Make-A-Voice employs a two-stage, decoder-only architecture that treats audio generation as a sequence prediction task.

The overall architecture of the Make-A-Voice system, illustrating the flow from prompts and IDs to the Multi-scale Voice LLM.

Discrete Representation and Modeling

The model operates on discrete tokens rather than raw waveforms. It utilizes semantic tokens (via XLSR-53) to encode linguistic meaning and acoustic tokens (via SoundStream codec) to capture vocal texture. The generation process occurs in two stages:

  • Semantic Modeling: Maps text or speech prompts to semantic tokens.
  • Acoustic Modeling: Transforms semantic tokens into acoustic tokens, conditioned on a 3-second reference "prompt" to maintain speaker identity.

Differentiable Multi-scale Transformer

To manage the long sequences inherent in high-fidelity audio, Make-A-Voice uses a hierarchical transformer. A Global Transformer processes audio "patches" to maintain long-range structural coherence, while a Local Transformer handles the fine-grained details within those patches. Finally, a unit-based vocoder (modified BigVGAN) reconstructs the 16kHz waveform.

Experimental Setup

The model was trained on a massive 200,000-hour dataset covering six languages. This includes rich-resource languages like English and Chinese, alongside low-resource languages (French, German, Spanish, and Japanese) which comprised only about 80 hours of the total data. The training objective encompassed four primary tasks: Text-to-Speech (TTS), Voice Conversion (VC), Singing Voice Synthesis (SVS), and Singing Voice Conversion (SVC).

Key Insights and Results

The study reveals several significant findings regarding the behavior of large-scale audio models:

  • Audio Scaling Laws: Increasing model size from 160M to 1.2B parameters resulted in a 37% improvement in TTS performance, confirming that scaling laws apply to generative audio.
  • Cross-Task Synergy: Joint training on speech and singing data significantly improved singing synthesis. The model showed a 27% improvement in style similarity (SIM) for singing when leveraging speech data.
  • Multilingual Scaffolding: Training on a mix of languages allowed the model to perform zero-shot voice cloning in low-resource languages more effectively than models trained on those languages in isolation, yielding a 20% gain in speaker similarity.
Performance comparison of Make-A-Voice against baseline models across various metrics including MOS, WER, and Speaker Similarity (SIM).

As shown in the results, Make-A-Voice achieved a Word Error Rate (WER) of 6.7 and a Speaker Similarity (SIM) of 0.85, outperforming competitive baselines like VALL-E and YourTTS in zero-shot scenarios.

Limitations and Future Work

Despite its performance, Make-A-Voice faces several challenges. Its output quality is inherently limited by the audio tokenizer; any information lost during discretization cannot be recovered. Furthermore, the model is currently focused on synthesis rather than a unified "understanding and generation" framework (e.g., including ASR). Future research will focus on improving tokenization robustness and exploring even more resource-constrained scenarios.

Summary updated

Abstract

Large language models (LLMs) have successfully served as a general-purpose interface across multiple tasks and languages, while the adaptation of voice LLMs is mostly designed for specific purposes (either single-task or monolingual), where the advantages of LLMs especially for low-resource language processing and zero-shot task generalization are less exploited in the audio community. To bridge the gap, we introduce Make-A-Voice as a multi-modal voice LLM and conduct a comprehensive study on its capability to deal with multiple tasks/languages. When trained on ~200K hours of 6-language data for 4 voice generation applications, Make-A-Voice emerges notable advantages: 1) as scalable learners to improve performance with end-to-end local and global multiscale transformers; and 2) as multitask learners by adjusting prompts to share common knowledge across modalities (speech/singing) and present in-context learning abilities by generalizing to unseen tasks not explicitly train on; 3) as multilingual learners to alleviate data scarcity of low-resource languages by including rich-resource language training data. Experimental results demonstrate that Make-A-Voice exhibits superior audio quality and style similarity compared with competitive baseline models in monolingual/cross-lingual voice generation. Audio samples are available at https://M-Voice.github.io
BibTeX
@inproceedings{huang2024make,
  author = {Rongjie Huang and Chunlei Zhang and Yongqi Wang and Dongchao Yang and Jinchuan Tian and Zhenhui Ye and Luping Liu and Zehan Wang and Ziyue Jiang and Xuankai Chang and Jiatong Shi and Chao Weng and Zhou Zhao and Dong Yu},
  title = {{Make-A-Voice: Revisiting Voice Large Language Models as Scalable Multilingual and Multitask Learners}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.589},
}
Figure