Abstract
Make-A-Voice is a unified, decoder-only voice Large Language Model (LLM) that treats various voice generation tasks—such as text-to-speech and singing synthesis—as a language modeling problem. By scaling to 1.2 billion parameters and training on 200,000 hours of multilingual data, it achieves state-of-the-art performance in zero-shot voice cloning. This research demonstrates that the "scaling hypothesis" holds for audio as it does for text, showing that larger models and diverse datasets significantly enhance audio quality and cross-task generalization.
Introduction
While text-based Large Language Models (LLMs) like GPT-4 have revolutionized natural language processing by serving as general-purpose interfaces, voice-based models have historically remained specialized. Existing "Voice LLMs" often excel in specific domains, such as English text-to-speech (TTS), but struggle with diverse tasks like singing synthesis or low-resource languages.
The authors of Make-A-Voice identify three critical gaps in the current landscape:
- Scalability: The impact of scaling model parameters and data volume on voice quality remains under-explored.
- Multitask Learning: Most models lack the ability to share knowledge across related audio tasks.
- Low-Resource Support: Data scarcity hinders the development of high-quality models for non-dominant languages.
Methods: A Voice-as-Language Approach
Make-A-Voice employs a two-stage, decoder-only architecture that treats audio generation as a sequence prediction task.
Discrete Representation and Modeling
The model operates on discrete tokens rather than raw waveforms. It utilizes semantic tokens (via XLSR-53) to encode linguistic meaning and acoustic tokens (via SoundStream codec) to capture vocal texture. The generation process occurs in two stages:
- Semantic Modeling: Maps text or speech prompts to semantic tokens.
- Acoustic Modeling: Transforms semantic tokens into acoustic tokens, conditioned on a 3-second reference "prompt" to maintain speaker identity.
Differentiable Multi-scale Transformer
To manage the long sequences inherent in high-fidelity audio, Make-A-Voice uses a hierarchical transformer. A Global Transformer processes audio "patches" to maintain long-range structural coherence, while a Local Transformer handles the fine-grained details within those patches. Finally, a unit-based vocoder (modified BigVGAN) reconstructs the 16kHz waveform.
Experimental Setup
The model was trained on a massive 200,000-hour dataset covering six languages. This includes rich-resource languages like English and Chinese, alongside low-resource languages (French, German, Spanish, and Japanese) which comprised only about 80 hours of the total data. The training objective encompassed four primary tasks: Text-to-Speech (TTS), Voice Conversion (VC), Singing Voice Synthesis (SVS), and Singing Voice Conversion (SVC).
Key Insights and Results
The study reveals several significant findings regarding the behavior of large-scale audio models:
- Audio Scaling Laws: Increasing model size from 160M to 1.2B parameters resulted in a 37% improvement in TTS performance, confirming that scaling laws apply to generative audio.
- Cross-Task Synergy: Joint training on speech and singing data significantly improved singing synthesis. The model showed a 27% improvement in style similarity (SIM) for singing when leveraging speech data.
- Multilingual Scaffolding: Training on a mix of languages allowed the model to perform zero-shot voice cloning in low-resource languages more effectively than models trained on those languages in isolation, yielding a 20% gain in speaker similarity.
As shown in the results, Make-A-Voice achieved a Word Error Rate (WER) of 6.7 and a Speaker Similarity (SIM) of 0.85, outperforming competitive baselines like VALL-E and YourTTS in zero-shot scenarios.
Limitations and Future Work
Despite its performance, Make-A-Voice faces several challenges. Its output quality is inherently limited by the audio tokenizer; any information lost during discretization cannot be recovered. Furthermore, the model is currently focused on synthesis rather than a unified "understanding and generation" framework (e.g., including ASR). Future research will focus on improving tokenization robustness and exploring even more resource-constrained scenarios.
Summary updated