An Independence-promoting Loss for Music Generation with Language Models

Jean-Marie Lemercier, Simon Rouard, Jade Copet, Yossi Adi, Alexandre Défossez

Abstract

State-of-the-art music generation often relies on language models (LMs) to predict discrete audio tokens from multiple codebooks. While parallel decoding strategies offer significant speedups, they assume statistical independence between codebooks—an assumption often violated by standard audio tokenizers. This paper introduces an "independence-promoting" loss function based on Maximum Mean Discrepancy (MMD) to regularize audio tokenizers. By enforcing statistical independence during training, the proposed method enables high-quality audio generation using fast, factorized decoding without the typical accuracy trade-offs. Experimental results on the MusicGen framework demonstrate significant improvements in both objective and subjective audio quality.

Introduction

Modern music generation models, such as MusicGen or AudioLM, typically employ a two-stage process: an auto-encoder compresses raw audio into multi-layer discrete tokens (codebooks), and a Transformer-based language model predicts these tokens. To achieve efficient inference, researchers often use factorized distributions, predicting codebooks in parallel or with a temporal delay. However, this approach assumes that codebooks are mutually independent. In practice, standard Residual Vector Quantization (RVQ) creates highly dependent layers, where deeper layers refine the information captured by earlier ones. This mismatch between the tokenizer's structure and the LM's decoding assumptions leads to degraded musical quality.

Methods

To address this, the authors propose an auxiliary objective, $L_{inde}$ (Independence Loss), applied during the training of the audio tokenizer.

The proposed architecture incorporates the independence-promoting loss $L_{inde}$ within the RVQ-based audio tokenizer, facilitating better alignment with the language model's decoding strategy.

The core of $L_{inde}$ is the Maximum Mean Discrepancy (MMD) in a Reproducing Kernel Hilbert Space (RKHS). MMD serves as a robust proxy for statistical independence by measuring the distance between the joint distribution of codebooks and the product of their marginal distributions. Unlike adversarial mutual information estimation, MMD is stable to train and easy to implement. The loss is specifically designed to match the "delay" pattern used during inference, effectively "uncoupling" latent variables by shuffling batch samples to simulate independent marginals. This approach is "plug-and-play," requiring no additional parameters or inference-time overhead.

Experimental Setup

The model was trained on a large-scale internal dataset comprising 20,000 hours of 32kHz licensed music. Evaluation was conducted using the MusicCaps benchmark, which contains 5.5k expert-curated samples. Additional analysis on FMA-Pop and LibriSpeech helped characterize mutual information behavior across different audio domains, such as music versus speech.

Results

Integrating the MMD-based loss into the MusicGen framework (MusicGen-MMD) yielded substantial performance gains across all metrics.

Comparison of MusicGen-MMD against baseline models across objective metrics (FAD, KL, CLAP) and subjective human ratings (OVRL).

As shown in the results table, MusicGen-MMD achieved a Frechet Audio Distance (FAD) of 2.98, outperforming the baseline MusicGen (3.60) and other competitive models like AudioLDM (3.52). Subjective MUSHRA-style listening tests further confirmed these findings, with human raters awarding MusicGen-MMD an overall quality score of 74.75, a significant jump from the baseline's 62.54. Crucially, these improvements were achieved while maintaining the high inference speed of factorized decoding, generating audio at the same frame rate as the auto-encoder.

Discussion and Conclusion

A key insight of this work is that improving the tokenizer's latent space to align with the Transformer's assumptions can yield "free" performance gains. The authors identified a trade-off between the natural hierarchy of RVQ and the desired independence; their MMD-based method allows for a tunable balance between these properties. While the current approach focuses on global independence, future work may explore conditional independence objectives. Additionally, addressing dataset biases—currently skewed toward Western music—remains a priority for improving the diversity and fairness of generative music models.

Summary updated

BibTeX
@inproceedings{lemercier2024independence,
  author = {Jean-Marie Lemercier and Simon Rouard and Jade Copet and Yossi Adi and Alexandre Défossez},
  title = {{An Independence-promoting Loss for Music Generation with Language Models}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=CLJZI5kDhX},
}
Figure