Abstract
UniAudio is a 1-billion-parameter foundation model that treats all audio generation—whether it's speech, music, or environmental sounds—as a unified "next-token prediction" task. By scaling to 165,000 hours of data and introducing a novel multi-scale Transformer architecture, it achieves state-of-the-art or competitive results across 11 diverse audio tasks with a single model.
Introduction
Traditionally, audio generation has been a fragmented field. Specialized models are typically developed for distinct tasks such as text-to-speech (TTS), voice conversion (VC), or music synthesis. These task-specific models rely heavily on domain-specific knowledge and are often restricted to fixed setups.
The challenge in creating a universal audio model lies in the sheer diversity of audio types—speech has different rhythmic and spectral patterns than a piano concerto or environmental sounds. Furthermore, modern neural codecs (which turn audio into discrete "tokens" for AI to process) often generate extremely long sequences. For a standard Transformer, these sequences are computationally expensive to process, leading to a bottleneck in both training and inference.
Methodology
UniAudio adopts the Large Language Model (LLM) paradigm but adapts it specifically for the complexities of sound. Its methodology rests on three pillars:
- Unified Tokenization: A universal neural codec based on Residual Vector Quantization (RVQ) maps any audio signal into a sequence of discrete tokens. Other modalities, such as text, phonemes, and MIDI, are similarly tokenized to create a unified input space.
- Task Formulation: Every task is formulated as a simple sequence:
[input conditions, target audio]. For example, a TTS task is represented as[phonemes + speaker prompt, speech tokens]. The model then learns to predict the next token in the sequence. - Multi-Scale Transformer: To address the computational complexity of long sequences, UniAudio employs a hierarchical architecture. A Global Transformer models inter-frame correlations (semantic level), while a Local Transformer models intra-frame correlations (acoustic details). This reduces complexity from $O((T \times n_q)^2)$ to $O(T^2)$, enabling efficient scaling.
Datasets and Training
The model was trained on a massive corpus of 165,000 hours of audio across 12 public datasets. This includes 80,000 hours of speech (LibriLight, MLS), as well as extensive sound, music, and singing data (AudioSet, Million Song Dataset, OpenCPOP). Evaluation was conducted on standard benchmarks like LibriSpeech and MusicCaps.
Key Results and Insights
UniAudio was tested on 11 tasks—7 during initial training and 4 via fine-tuning—delivering impressive performance across the board:
- Speech Synthesis (TTS): Achieved a Word Error Rate (WER) of 2.0, outperforming many state-of-the-art (SOTA) baselines.
- Voice Conversion: Reached a speaker similarity score (SIM) of 0.87, demonstrating superior voice cloning abilities.
- Zero-Shot Capability: In subjective evaluations (MOS), UniAudio consistently ranked at the top for speech quality and naturalness, even for speakers not seen during training.
- Multi-Task Synergy: Training on multiple tasks simultaneously was found to be more effective than single-task training. The model learns shared intrinsic properties of audio across domains.
Conclusion and Future Work
UniAudio demonstrates that a single foundation model can effectively handle a wide array of audio generation tasks. By splitting modeling into global and local scales, high-quality codecs can be used without prohibitive memory requirements. Future work will look toward expanding task coverage, incorporating unlabeled data, and adding new modalities like video or image conditions to further enhance the model's understanding of sound.
Summary updated