UniAudio: An Audio Foundation Model Toward Universal Audio Generation

Dongchao Yang, Jinchuan Tian, Xu Tan, Rongjie Huang, Songxiang Liu, Xuankai Chang, Jiatong Shi, Sheng Zhao, Jiang Bian, Xixin Wu, Zhou Zhao, Shinji Watanabe, Helen Meng

Abstract

UniAudio is a 1-billion-parameter foundation model that treats all audio generation—whether it's speech, music, or environmental sounds—as a unified "next-token prediction" task. By scaling to 165,000 hours of data and introducing a novel multi-scale Transformer architecture, it achieves state-of-the-art or competitive results across 11 diverse audio tasks with a single model.

Introduction

Traditionally, audio generation has been a fragmented field. Specialized models are typically developed for distinct tasks such as text-to-speech (TTS), voice conversion (VC), or music synthesis. These task-specific models rely heavily on domain-specific knowledge and are often restricted to fixed setups.

The challenge in creating a universal audio model lies in the sheer diversity of audio types—speech has different rhythmic and spectral patterns than a piano concerto or environmental sounds. Furthermore, modern neural codecs (which turn audio into discrete "tokens" for AI to process) often generate extremely long sequences. For a standard Transformer, these sequences are computationally expensive to process, leading to a bottleneck in both training and inference.

Methodology

UniAudio adopts the Large Language Model (LLM) paradigm but adapts it specifically for the complexities of sound. Its methodology rests on three pillars:

  1. Unified Tokenization: A universal neural codec based on Residual Vector Quantization (RVQ) maps any audio signal into a sequence of discrete tokens. Other modalities, such as text, phonemes, and MIDI, are similarly tokenized to create a unified input space.
  2. Task Formulation: Every task is formulated as a simple sequence: [input conditions, target audio]. For example, a TTS task is represented as [phonemes + speaker prompt, speech tokens]. The model then learns to predict the next token in the sequence.
  3. Multi-Scale Transformer: To address the computational complexity of long sequences, UniAudio employs a hierarchical architecture. A Global Transformer models inter-frame correlations (semantic level), while a Local Transformer models intra-frame correlations (acoustic details). This reduces complexity from $O((T \times n_q)^2)$ to $O(T^2)$, enabling efficient scaling.
UniAudio framework and Multi-Scale Transformer architecture.

Datasets and Training

The model was trained on a massive corpus of 165,000 hours of audio across 12 public datasets. This includes 80,000 hours of speech (LibriLight, MLS), as well as extensive sound, music, and singing data (AudioSet, Million Song Dataset, OpenCPOP). Evaluation was conducted on standard benchmarks like LibriSpeech and MusicCaps.

Key Results and Insights

UniAudio was tested on 11 tasks—7 during initial training and 4 via fine-tuning—delivering impressive performance across the board:

  • Speech Synthesis (TTS): Achieved a Word Error Rate (WER) of 2.0, outperforming many state-of-the-art (SOTA) baselines.
  • Voice Conversion: Reached a speaker similarity score (SIM) of 0.87, demonstrating superior voice cloning abilities.
  • Zero-Shot Capability: In subjective evaluations (MOS), UniAudio consistently ranked at the top for speech quality and naturalness, even for speakers not seen during training.
  • Multi-Task Synergy: Training on multiple tasks simultaneously was found to be more effective than single-task training. The model learns shared intrinsic properties of audio across domains.
Performance comparison of UniAudio against state-of-the-art baselines across multiple audio tasks.

Conclusion and Future Work

UniAudio demonstrates that a single foundation model can effectively handle a wide array of audio generation tasks. By splitting modeling into global and local scales, high-quality codecs can be used without prohibitive memory requirements. Future work will look toward expanding task coverage, incorporating unlabeled data, and adding new modalities like video or image conditions to further enhance the model's understanding of sound.

Summary updated

Abstract

Language models (LMs) have demonstrated the capability to handle a variety of generative tasks. This paper presents the UniAudio system, which, unlike prior task-specific approaches, leverages LMs techniques to generate multiple types of audio (including speech, sounds, music, and singing) with given input conditions. UniAudio 1) first tokenizes all types of target audio along with other condition modalities, 2) concatenates source-target pair as a single sequence, and 3) performs next-token prediction using LMs. Also, a multi-scale Transformer model is proposed to handle the overly long sequences caused by the residual vector quantization based neural codec in tokenization. Training of UniAudio is scaled up to 165K hours of audio and 1B parameters, based on all generative tasks, aiming to obtain sufficient prior knowledge not only in the intrinsic properties of audio but also the inter-relationship between audio and other modalities. Therefore, the trained UniAudio model has the potential to become a foundation model for universal audio generation: it shows strong capability in all trained tasks and can seamlessly support new audio generation tasks after simple fine-tuning. Experiments demonstrate that UniAudio achieves state-of-the-art or at least competitive results on most of the 11 tasks. Demo and code are released\footnote{\url{https://uniaudio666.github.io/demo_UniAudio/}}
BibTeX
@misc{yang2023uniaudio,
  author = {Dongchao Yang and Jinchuan Tian and Xu Tan and Rongjie Huang and Songxiang Liu and Xuankai Chang and Jiatong Shi and Sheng Zhao and Jiang Bian and Xixin Wu and Zhou Zhao and Shinji Watanabe and Helen Meng},
  title = {{UniAudio: An Audio Foundation Model Toward Universal Audio Generation}},
  year = {2023},
  eprint = {2310.00704},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2310.00704},
}
Figure