XMAD-Bench: Cross-Domain Multilingual Audio Deepfake Benchmark

Ioan-Paul Ciobanu, Andrei-Iulian Hiji, Nicolae-Catalin Ristea, Paul Irofti, Cristian Rusu, Radu Tudor Ionescu

TL;DR

While state-of-the-art audio deepfake detectors boast near-perfect accuracy on paper, they fail dramatically when deployed "in the wild" against unseen generative models and languages. To bridge this gap, the authors introduce XMAD-Bench, a massive 668.8-hour cross-domain, multilingual dataset designed to rigorously test how well deepfake detectors generalize across distinct speakers, languages, and synthesis methods.

What Problem Does This Paper Address?

Most existing audio deepfake detection models report accuracy rates close to 99%. However, these impressive figures are usually the result of an in-domain evaluation flaw: the models are trained and tested on deepfakes generated by the exact same algorithms and text-to-speech (TTS) tools.

In the real world, bad actors constantly rotate through new, unknown voice-cloning architectures. Furthermore, most public deepfake datasets are strictly English-only or monolingual, which introduces severe language biases into detectors. This paper tackles the generalization crisis in audio deepfake detection by providing a benchmark that forces models to operate under true cross-domain and multilingual conditions.

Methods

The authors constructed a rigorous pipeline to generate high-quality, realistic deepfakes. For any given real audio clip, the text transcript is first passed to a TTS model, and the resulting audio is then processed by a Voice Conversion (VC) tool to clone the original speaker's unique vocal characteristics.

Figure 2: General flow for fake sample generation based on various text-to-speech and voice conversion tools.

To evaluate current defense capabilities, the researchers benchmarked six diverse neural architectures:

  • Convolutional Models: ResNet-18 and ResNet-50.
  • Spectrogram Transformers: Audio Spectrogram Transformer (AST) and SepTr (Separable Transformer).
  • Self-Supervised Foundation Models: wav2vec 2.0 and a frozen Whisper-Large-v3 encoder paired with a shallow Multi-Layer Perceptron (MLP).

Crucially, the dataset is split such that the generative methods, speakers, and real audio sources used in the cross-domain test set are entirely unseen during training.

Datasets

The authors introduce XMAD-Bench (Cross-Domain Multilingual Audio Deepfake Benchmark), which comprises 668.8 hours of speech and is balanced perfectly with 207K real and 207K fake samples.

Figure 1: XMAD-Bench comprises 668.8 hours of real and fake speech samples across seven languages: Arabic (Ar), English (En), German (De), Mandarin Chinese (Zh), Romanian (Ro), Russian (Ru), and Spanish (Es). For each language, there are two sources of real samples, enabling us to organize the dataset in a cross-domain format. Best viewed in color.

The dataset spans seven languages: Arabic, English, German, Mandarin Chinese, Romanian, Russian, and Spanish. Real audio samples were curated from diverse open-source repositories including Common Voice (for training/in-domain testing) and MASC, M-AILABS, AISHELL-3, and VoxPopuli (for cross-domain testing). Fake samples were generated using a wide array of tools like XTTSv2, YourTTS, VALL-E-X, Bark, and OpenVoice.

Key Insights and Ideas

  • The Generalization Illusion: Models that achieve a flawless 100% accuracy on in-domain data frequently collapse to performance levels worse than a coin flip when facing unseen generative models.
  • Model-Specific Biases Over Robust Learning: Rather than learning the intrinsic acoustic signatures of "fakeness," standard classifiers often latch onto dataset-specific shortcuts. For example, when tested on Arabic cross-domain data, the AST model developed a heavy bias toward labeling almost all samples as fake, while ResNet-18 suffered from high rates of both false positives and false negatives.
Figure 3: Cross-domain confusion matrices of ResNet- 18 (first row) and AST (second row) on Arabic (first column) and English (second column). Best viewed in color.
  • Foundation Models Move the Needle: Self-supervised speech representations (wav2vec 2.0 and Whisper) generalize significantly better than models trained from scratch on spectrograms. Their pre-training on massive, diverse audio corpora helps them capture language-independent features of synthetic speech.

Key Results

The quantitative drop-off from in-domain to cross-domain testing is stark:

  • In-Domain: ResNet-18, ResNet-50, and wav2vec 2.0 all achieved 100% accuracy and a 0% Equal Error Rate (EER) across nearly all languages.
  • Cross-Domain Collapse: On Arabic cross-domain data, ResNet-50's accuracy plummeted to 25.09% (with an EER of 68.50%).
  • The Winners: The Whisper+MLP baseline proved to be the most robust, achieving an impressive 90.94% accuracy and an 8.38% EER on the challenging English cross-domain split, vastly outperforming standard CNNs.
  • Noise Robustness: When subjected to background Gaussian noise ($\sigma = 0.01$), the Whisper-based model maintained a stable accuracy of 80.77%, proving that foundation model backbones provide a buffer against real-world acoustic distortions.

Limitations and Future Work

A primary limitation acknowledged by the authors is that due to varying open-source tool support across different languages, they could not use the exact same generative models across all seven languages. Additionally, this paper focuses purely on benchmarking the current state of the art; the authors did not implement or evaluate explicit domain adaptation techniques to mitigate the performance drops. They leave the development of robust domain-adaptive audio deepfake detectors as a crucial direction for future research.

Summary updated

Abstract

Recent advances in audio generation led to an increasing number of deepfakes, making the general public more vulnerable to financial scams, identity theft, and misinformation. Audio deepfake detectors promise to alleviate this issue, with many recent studies reporting accuracy rates close to 99%. However, these methods are typically tested in an in-domain setup, where the deepfake samples from the training and test sets are produced by the same generative models. To this end, we introduce XMAD-Bench, a large-scale cross-domain multilingual audio deepfake benchmark comprising 668.8 hours of real and deepfake speech. In our novel dataset, the speakers, the generative methods, and the real audio sources are distinct across training and test splits. This leads to a challenging cross-domain evaluation setup, where audio deepfake detectors can be tested "in the wild". Our in-domain and cross-domain experiments indicate a clear disparity between the in-domain performance of deepfake detectors, which is usually as high as 100%, and the cross-domain performance of the same models, which is sometimes similar to random chance. Our benchmark highlights the need for the development of robust audio deepfake detectors, which maintain their generalization capacity across different languages, speakers, generative methods, and data sources. Our benchmark is publicly released at https://github.com/ristea/xmad-bench/.
BibTeX
@inproceedings{ciobanu2026xmad,
  author = {Ioan-Paul Ciobanu and Andrei-Iulian Hiji and Nicolae-Catalin Ristea and Paul Irofti and Cristian Rusu and Radu Tudor Ionescu},
  title = {{XMAD-Bench: Cross-Domain Multilingual Audio Deepfake Benchmark}},
  booktitle = {EACL (Findings)},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.162},
}
Figure