Abstract
As AI-generated music becomes increasingly prevalent on streaming platforms, the need for robust detection methods has grown critical. Traditional audio-based detectors, while accurate on known datasets, often fail when faced with audio perturbations or unseen generation models. This paper, a collaboration between Deezer Research and the Johannes Kepler University Linz, introduces a novel detection pipeline that focuses on linguistic patterns in lyrics rather than audio artifacts. By transcribing audio into text and utilizing Large Language Model (LLM) embeddings, the proposed method achieves high accuracy and demonstrates superior robustness and generalization compared to state-of-the-art audio classifiers.
Introduction
The music industry is currently navigating a surge in AI-generated content. Recent data from Deezer suggests that approximately 10% of daily uploads are now generated by artificial intelligence. This shift presents significant challenges for copyright enforcement, royalty distribution, and platform transparency. While detection tools exist, they are often easily bypassed by simple audio processing or the release of new generative models. This research explores a more stable signal for detection: the lyrical content of the songs.
The Limitations of Audio-Based Detection
Existing detection methods generally fall into two categories: audio-based and lyrics-based. Audio-based detectors analyze low-level digital "fingerprints" or artifacts left by neural audio decoders. While these models can achieve over 99% accuracy on specific datasets, they are notoriously "brittle." Minor changes to a track—such as pitch shifting, equalization adjustments, or the addition of background noise—can cause performance to plummet. Furthermore, these models often fail to generalize to new AI generators (e.g., Suno or Udio) that the detector was not specifically trained on.
Lyrics-based detection offers a promising alternative, as the stylistic "AI-speak" remains relatively consistent across different audio rendering engines. However, previous lyrics-based approaches required clean, ground-truth text from databases, which is rarely available for raw audio uploads. This paper bridges that gap by integrating Automated Speech Recognition (ASR) directly into the detection pipeline.
Proposed Methodology
The researchers developed a three-stage pipeline designed to identify AI-generated songs directly from raw audio files:
- Transcription: The system uses Whisper-large-v2 to convert audio into raw text transcripts. A key finding of the study is that the detector does not require perfect transcription; even "noisy" transcripts containing ASR errors provide sufficient linguistic data for accurate classification.
- Feature Extraction: The resulting transcript is processed through a pre-trained text encoder. The researchers compared several models, including retrieval-optimized (BGE-M3) and stylistic (UAR) encoders. The best performance was achieved using LLM2Vec (based on Llama-3), which transforms a standard LLM into a bidirectional encoder capable of capturing deep semantic and syntactic nuances.
- Classification: A simple Multi-Layer Perceptron (MLP) is trained on these embeddings to classify the track as either "Real" or "Fake."
Experimental Setup and Results
The study utilized a diverse, multi-lingual corpus comprising 3,704 real and 3,558 AI-generated lyrics across nine languages and six genres. To create a realistic testing environment, the researchers used Suno v3.5 to generate 4-minute songs from the AI-generated lyrics. To test generalization, they also created a separate set of 260 songs using Udio v1.5, a model entirely unseen by the detector during training.
Key Performance Metrics
- High Accuracy: The Whisper + LLM2Vec-LLaMA model achieved a 90.7% macro-recall across all languages.
- Resilience to Noise: There was less than a 4% performance gap between using perfect "booklet" lyrics and raw, uncorrected ASR transcripts, proving the method's viability for real-world streaming catalogs.
- Robustness to Attacks: When subjected to audio perturbations like pitch shifting and reverb, the lyrics-based model remained stable. In contrast, a standard CNN-based audio detector saw its recall drop to as low as 59% under similar conditions.
- Generalization: When tested on the unseen Udio generator, the audio-based detector failed (56.9% recall), while the lyrics-based model maintained a strong 85.9% recall.
Discussion and Limitations
The core insight of this research is that lyrics serve as an "invariant" signal. While audio generation technology evolves rapidly, the underlying linguistic patterns of AI-generated text remain detectable. However, the model does face challenges. It struggled with specific niche combinations, such as Italian Jazz or Turkish Folk, likely due to limited training data in those areas. Additionally, the current system is binary; it does not yet account for hybrid content, such as a human artist performing AI-written lyrics or an AI-generated track using human-written lyrics.
Conclusion
By shifting the focus from audio artifacts to linguistic style, this research provides a more robust and future-proof method for identifying AI-generated music. The ability to maintain high accuracy despite audio noise and across different generation models marks a significant step forward for platform integrity. Future work will likely explore multi-modal fusion, combining both audio and lyrical analysis to create an even more comprehensive detection system.
Summary updated