TL;DR
The paper introduces FakeMusicCaps, a new benchmark dataset designed to advance the field of audio forensics in detecting and attributing AI-generated music. By evaluating standard deep learning baselines, the authors demonstrate that while identifying known open-source text-to-music generators is currently trivial, detecting high-quality, unknown commercial generators remains a significant open challenge.
What Problem Does This Paper Address?
The rapid rise of Text-to-Music (TTM) models like Suno and Udio has democratized music production, but it has also triggered massive copyright and attribution controversies. While the multimedia forensics community has well-established tools for detecting deepfake images, videos, and synthetic speech, music deepfake detection remains largely uncharted territory.
The core challenge is two-fold:
- Detection: Is a piece of music real or synthetic?
- Attribution: If it is synthetic, which specific AI model generated it?
To solve these problems, researchers need standardized datasets. This paper addresses this gap by introducing a dataset specifically tailored for TTM detection and attribution, evaluating it under both "closed-set" (all models known) and "open-set" (encountering an unknown model) scenarios.
Methods
The authors frame the problem as a multi-class classification task. They evaluate three baseline neural network architectures:
- M5: A relatively simple, lightweight convolutional neural network (CNN) that operates directly on raw audio waveforms.
- RawNet2: A more complex end-to-end architecture designed for anti-spoofing, utilizing Sinc-filters and Gated Recurrent Units (GRUs) on raw audio.
- ResNet18 + Spec: A standard 18-layer ResNet architecture modified to process 2D log-spectrograms (time-frequency representations) of the audio.
For the more realistic open-set scenario, where the system encounters a generator it wasn't trained on, they test two classification strategies:
- Thresholding: Comparing the ratio of the top two predicted class probabilities. If the network is highly uncertain between multiple known classes, the sample is flagged as "unknown."
- One-Class SVM: Training a Support Vector Machine on the output probabilities of the known classes to detect out-of-distribution samples.
Datasets
The paper introduces FakeMusicCaps, which is built on top of the widely used MusicCaps dataset (5.5k professional text-music pairs). The authors took the original text captions from MusicCaps and fed them into five state-of-the-art, open-source TTM models to generate 10-second audio clips:
- MusicGen
- MusicLDM
- AudioLDM2
- Stable Audio Open
- Mustango
This yielded a total of 27,605 synthetic tracks (~77 hours of audio), all standardized to 16 kHz mono WAV format.
To test the open-set performance, they used a separate dataset called SunoCaps, which contains music generated by the popular, closed-source commercial platform Suno. Suno acted as the "unknown" generator during testing.
Key Insights and Ideas
- Open-source models leave distinct fingerprints: In a closed-set scenario, the spectrogram-based model achieved perfect accuracy. This indicates that current open-source TTM architectures leave unique acoustic artifacts that deep learning models can easily exploit for attribution.
- The "Too Realistic to Detect" Trap: In open-set testing using the thresholding method, the top-performing model (
ResNet18 + Spec) misclassified all unknown Suno-generated clips as real music. Because commercial models like Suno generate incredibly high-fidelity audio, they successfully fool detectors trained only on open-source fakes. - Window size has minimal impact: Shifting the input window size from 10 seconds down to 2.5 seconds resulted in only minor drops in accuracy. This suggests that the synthetic "fingerprints" are distributed uniformly throughout the tracks, rather than relying on long-term musical structure.
Key Results
- Closed-Set Attribution:
ResNet18 + Specachieved a flawless 1.00 Balanced Accuracy (ACC_B), outperforming M5 (0.90) and RawNet2 (0.88). - Open-Set (Thresholding):
ResNet18 + Specmaintained the highest overall accuracy (0.85 ACC_B), but completely failed to isolate the unknown class, confusing it entirely with real audio. - Open-Set (SVM): Performance plummeted across all models, with balanced accuracies dropping to between 0.42 and 0.48, highlighting how poorly standard out-of-distribution methods handle this task.
Limitations and Future Work
The primary limitation acknowledged by the authors is that FakeMusicCaps is built entirely from open-source generators (with commercial ones only used for testing). As commercial models rapidly evolve, detectors will need to adapt to even cleaner audio signals.
For future work, the authors propose two exciting directions:
- Exploiting Text-Music Alignment: Investigating whether the semantic relationship between the text prompt and the generated audio can be used as a forensic feature.
- Musicology-Informed Forensics: Leveraging music theory (e.g., detecting unnatural chord progressions, tempo drifts, or key changes) to spot deepfakes, which could be especially useful for detecting "splicing" attacks where AI music is mixed into real tracks.
Summary updated