A Tutorial on Generative Models for Music Audio

Mateusz Modrzejewski, Joanna Kurkowska, Dorien HerremansarXiv 2026 · arXiv

TL;DR

This comprehensive tutorial provides a technical roadmap for the generative music audio landscape, tracing the field's rapid evolution from the landmark models of early 2023 to the sophisticated song-generation and real-time systems of late 2026. It establishes a shared notation for the two dominant architectural lineages—codec-token language models and latent diffusion—while addressing the critical challenges of evaluation, data provenance, and legal compliance.

What Problem Does This Paper Address?

Generating music audio is significantly more complex than generating speech or images. It requires maintaining long-range structural coherence over minutes, precise rhythmic timing, and the intricate coordination of vocals, lyrics, and accompaniment.

Since the breakthrough of models like MusicLM in 2023, the literature has become fragmented, with different papers using inconsistent notation and evaluation protocols. Furthermore, standard metrics like the Fréchet Audio Distance (FAD) have been shown to correlate weakly with actual human musical preference. This monograph synthesizes these scattered developments into a single pedagogical framework, helping researchers and engineers navigate the trade-offs between fidelity, sequence length, and computational cost.

Methods

The authors categorize music generation into five principal generative paradigms:

  1. Autoregressive Token Models: Predicting discrete audio tokens (from codecs like EnCodec or DAC) one by one or in delayed parallel streams (e.g., MusicGen).
  2. Masked/Parallel Decoding: Predicting many tokens at once and refining them over several rounds to reduce latency.
  3. Latent Diffusion: Starting from noise in a compressed continuous space and iteratively denoising it (e.g., Stable Audio).
  4. Flow Matching: Regressing onto a velocity field along a path from noise to data, often allowing for fewer sampling steps than diffusion.
  5. Planner-Renderer Hybrids: A "best of both worlds" approach where a language model plans a coarse structure (lyrics, melody, or sketch) and a diffusion/flow model renders the high-fidelity audio.

The paper also details Preference Alignment techniques (like DPO and RLHF) used to steer models toward human-preferred outputs and Sampling Acceleration methods (like consistency models and distillation) that enable faster-than-real-time generation.

Datasets

The monograph highlights a major transparency gap: most industrial training corpora (e.g., Suno, Udio, Qwen-Music) remain undisclosed. However, it references several key open resources:

  • Training/Captions: MusicCaps, Song Describer, and JamendoMaxCaps (large-scale captions for freely licensed music).
  • Benchmarks: MusicPrefs (pairwise human preferences), SongBench (multi-aspect quality for full songs), and ZoME-Bench (for music editing).
  • Evaluation: The authors emphasize the move toward MAUVE Audio Divergence (MAD) and learned judges to replace the aging FAD metric.

Key Insights and Ideas

  • The Convergence of Lineages: The field is moving away from "pure" token or diffusion models toward Planner-Renderer hybrids. This separation of concerns allows the model to handle long-range structure in the discrete stage and acoustic detail in the continuous stage.
  • The "Exposure Bias" in Planning: A major hurdle for hybrid models is that the renderer is often trained on "clean" plans from real data but must handle "imperfect" plans generated by the model at inference time.
  • Metric Fallibility: FAD is highly sensitive to the choice of embedding and reference set. A system can "game" FAD without actually sounding better to human listeners.
  • Legal Battlegrounds: The authors highlight that "memorization"—the model's ability to reproduce training data—is a central technical and legal question, as seen in recent court judgments in Munich and the U.S.

Key Results

  • Speedup: Through distillation and flow matching, models have moved from sampling times measured in hours (Jukebox) to faster-than-real-time and streaming capabilities (e.g., Presto, Stable Audio 3).
  • Scale: Industrial models are now trained on upwards of 5 million hours of audio, whereas academic baselines typically use tens of thousands of hours.
  • Song Length: Generation has expanded from 30-second instrumental clips to full 5-minute songs with coherent verse-chorus structures and intelligible lyrics (e.g., YuE, Suno v6).

Limitations and Future Work

  • Evaluation Gaps: There is still no universally accepted protocol for evaluating full songs (lyric intelligibility, vocal quality, and arrangement) or long-range form over several minutes.
  • Diversity vs. Alignment: Post-training for preference often reduces the diversity of a model's output, a trade-off that remains an open research problem.
  • Cultural Bias: Current models are heavily skewed toward English and Mandarin popular music. Most of the world's musical traditions remain under-represented in documented training data.
  • Interpretability: Little is known about the internal "concepts" music models learn; the authors suggest that finding and steering these concepts (via sparse autoencoders) is a promising path for more controllable generation.

Summary updated

Abstract

Music audio generation spans instrumental clips, full songs with vocals, edits of existing recordings, and systems that play alongside a musician in real time. This monograph explains the methods behind these systems for readers familiar with neural-network training who are new to music generation. It introduces neural audio codecs and autoencoders, autoregressive and masked token models, diffusion and flow matching, sampling acceleration, and preference optimization in a shared notation. Worked examples connect training objectives and sampling procedures to published systems. The application sections relate the musical requirements of instrumental generation, songs, editing, and live performance to model design. Comparison tables record representations, methods, inputs, tasks, and release information documented in papers and official sources. Symbolic composition, speech, and sound effects are included where they connect to music audio. We explain how to interpret automatic metrics and learned judges, design and report listening tests, and assess evidence about data provenance, memorization, attribution, and output detection. Selected legal decisions and disputes illustrate the implications for training and release. The reviewed systems combine established generative paradigms with new representations, planning stages, and post-training procedures. Differences in data, scale, and evaluation limit what cross-system comparisons can establish about the sources of reported gains.
BibTeX
@misc{modrzejewski2026tutorial,
  author = {Mateusz Modrzejewski and Joanna Kurkowska and Dorien Herremans},
  title = {{A Tutorial on Generative Models for Music Audio}},
  year = {2026},
  eprint = {2610.38252},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.38252},
}
Figure