FakeMusicCaps: a Dataset for Detection and Attribution of Synthetic Music Generated via Text-to-Music Models

Luca Comanducci, Paolo Bestagini, Stefano TubaroarXiv 2024 · arXiv

TL;DR

The paper introduces FakeMusicCaps, a new benchmark dataset designed to advance the field of audio forensics in detecting and attributing AI-generated music. By evaluating standard deep learning baselines, the authors demonstrate that while identifying known open-source text-to-music generators is currently trivial, detecting high-quality, unknown commercial generators remains a significant open challenge.

What Problem Does This Paper Address?

The rapid rise of Text-to-Music (TTM) models like Suno and Udio has democratized music production, but it has also triggered massive copyright and attribution controversies. While the multimedia forensics community has well-established tools for detecting deepfake images, videos, and synthetic speech, music deepfake detection remains largely uncharted territory.

The core challenge is two-fold:

  1. Detection: Is a piece of music real or synthetic?
  2. Attribution: If it is synthetic, which specific AI model generated it?

To solve these problems, researchers need standardized datasets. This paper addresses this gap by introducing a dataset specifically tailored for TTM detection and attribution, evaluating it under both "closed-set" (all models known) and "open-set" (encountering an unknown model) scenarios.

Fig. 1: Schematic representation of the text-to-music attribution problem.

Methods

The authors frame the problem as a multi-class classification task. They evaluate three baseline neural network architectures:

  • M5: A relatively simple, lightweight convolutional neural network (CNN) that operates directly on raw audio waveforms.
  • RawNet2: A more complex end-to-end architecture designed for anti-spoofing, utilizing Sinc-filters and Gated Recurrent Units (GRUs) on raw audio.
  • ResNet18 + Spec: A standard 18-layer ResNet architecture modified to process 2D log-spectrograms (time-frequency representations) of the audio.

For the more realistic open-set scenario, where the system encounters a generator it wasn't trained on, they test two classification strategies:

  1. Thresholding: Comparing the ratio of the top two predicted class probabilities. If the network is highly uncertain between multiple known classes, the sample is flagged as "unknown."
  2. One-Class SVM: Training a Support Vector Machine on the output probabilities of the known classes to detect out-of-distribution samples.

Datasets

The paper introduces FakeMusicCaps, which is built on top of the widely used MusicCaps dataset (5.5k professional text-music pairs). The authors took the original text captions from MusicCaps and fed them into five state-of-the-art, open-source TTM models to generate 10-second audio clips:

  • MusicGen
  • MusicLDM
  • AudioLDM2
  • Stable Audio Open
  • Mustango

This yielded a total of 27,605 synthetic tracks (~77 hours of audio), all standardized to 16 kHz mono WAV format.

To test the open-set performance, they used a separate dataset called SunoCaps, which contains music generated by the popular, closed-source commercial platform Suno. Suno acted as the "unknown" generator during testing.

Key Insights and Ideas

  • Open-source models leave distinct fingerprints: In a closed-set scenario, the spectrogram-based model achieved perfect accuracy. This indicates that current open-source TTM architectures leave unique acoustic artifacts that deep learning models can easily exploit for attribution.
  • The "Too Realistic to Detect" Trap: In open-set testing using the thresholding method, the top-performing model (ResNet18 + Spec) misclassified all unknown Suno-generated clips as real music. Because commercial models like Suno generate incredibly high-fidelity audio, they successfully fool detectors trained only on open-source fakes.
  • Window size has minimal impact: Shifting the input window size from 10 seconds down to 2.5 seconds resulted in only minor drops in accuracy. This suggests that the synthetic "fingerprints" are distributed uniformly throughout the tracks, rather than relying on long-term musical structure.

Key Results

  • Closed-Set Attribution: ResNet18 + Spec achieved a flawless 1.00 Balanced Accuracy (ACC_B), outperforming M5 (0.90) and RawNet2 (0.88).
  • Open-Set (Thresholding): ResNet18 + Spec maintained the highest overall accuracy (0.85 ACC_B), but completely failed to isolate the unknown class, confusing it entirely with real audio.
  • Open-Set (SVM): Performance plummeted across all models, with balanced accuracies dropping to between 0.42 and 0.48, highlighting how poorly standard out-of-distribution methods handle this task.

Limitations and Future Work

The primary limitation acknowledged by the authors is that FakeMusicCaps is built entirely from open-source generators (with commercial ones only used for testing). As commercial models rapidly evolve, detectors will need to adapt to even cleaner audio signals.

For future work, the authors propose two exciting directions:

  1. Exploiting Text-Music Alignment: Investigating whether the semantic relationship between the text prompt and the generated audio can be used as a forensic feature.
  2. Musicology-Informed Forensics: Leveraging music theory (e.g., detecting unnatural chord progressions, tempo drifts, or key changes) to spot deepfakes, which could be especially useful for detecting "splicing" attacks where AI music is mixed into real tracks.

Summary updated

Abstract

Text-To-Music (TTM) models have recently revolutionized the automatic music generation research field. Specifically, by reaching superior performances to all previous state-of-the-art models and by lowering the technical proficiency needed to use them. Due to these reasons, they have readily started to be adopted for commercial uses and music production practices. This widespread diffusion of TTMs poses several concerns regarding copyright violation and rightful attribution, posing the need of serious consideration of them by the audio forensics community. In this paper, we tackle the problem of detection and attribution of TTM-generated data. We propose a dataset, FakeMusicCaps that contains several versions of the music-caption pairs dataset MusicCaps re-generated via several state-of-the-art TTM techniques. We evaluate the proposed dataset by performing initial experiments regarding the detection and attribution of TTM-generated audio.
BibTeX
@misc{comanducci2024fakemusiccaps,
  author = {Luca Comanducci and Paolo Bestagini and Stefano Tubaro},
  title = {{FakeMusicCaps: a Dataset for Detection and Attribution of Synthetic Music Generated via Text-to-Music Models}},
  year = {2024},
  eprint = {2409.10684},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2409.10684},
}

Loading related content...

Figure