When Normality Shifts: Multi-Level WavLM Modeling for Open-World Audio Deepfake Detection

Daniel Niekurzak, Daniel Kozuch, Weronika Folwarska, Khan Muhammad, Rafał SchererarXiv 2026 · arXiv

TL;DR

This paper investigates "attack-free" audio deepfake detection by modeling the distribution of authentic speech using frozen WavLM features. While this normality-based approach nearly matches supervised models in controlled environments, it struggles significantly with real-world domain shifts (like compression), revealing that simply adjusting decision thresholds is insufficient for robust open-world deployment.

What Problem Does This Paper Address?

Most audio deepfake detectors are supervised: they are trained on specific examples of both real (bona fide) and fake (spoofed) speech. The danger is that these models learn "shortcuts"—artifacts unique to specific AI generators or recording setups—that fail when a new, unseen synthesis method emerges.

This is an open-world problem. The authors explore an alternative: Anomaly Detection. Instead of learning what a "fake" sounds like, can we build a robust model of what "real" speech looks like and flag anything that deviates from it? This "attack-free" protocol is harder but theoretically more generalizable to future, unknown deepfake techniques.

Methods

The authors propose MLN-WavLM (Multi-Level Normality WavLM). The pipeline is intentionally simple to isolate the effectiveness of the representations:

  1. Feature Extraction: They use a frozen WavLM Base+ model, a large-scale self-supervised speech encoder. They extract hidden representations from four different depths (layers 3, 6, 9, and 12).
  2. Normality Modeling: For each layer, they fit a shrinkage-Mahalanobis model using only bona fide speech from the ASVspoof 2019 LA dataset. This creates a statistical "envelope" of authentic speech.
  3. Scoring: Synthetic audio is scored based on its distance from this authentic distribution. The final score is an equal-weight ensemble of the distances from all four layers.
  4. Threshold Adaptation: To simulate real-world deployment, they tested methods to re-estimate the decision threshold from unlabeled incoming audio streams, comparing naive quantiles against robust statistics (Median/MAD).

Crucially, no spoofed samples were used to train the model, select features, or set the initial operating thresholds.

Datasets

  • ASVspoof 2019 LA (Logical Access): Used as the "source domain" for fitting the normality model and in-domain evaluation.
  • ASVspoof 2021 DF (Deepfake): Used as the "cross-domain" test set. This dataset is much more challenging as it includes diverse synthesis systems and, importantly, lossy compression (codecs), which shifts the distribution of both real and fake audio.

Key Insights and Ideas

  • Early Layers are Key: Layer-wise analysis (see Figure 2) showed that early WavLM representations (specifically Layer 3) carry the most discriminative signal for deepfake detection. Later layers added very little complementary information.
  • The Fragility of "Normal": A major finding is that the "normality" of speech is highly sensitive to the acoustic environment and codecs. A threshold calibrated on clean, high-quality speech (2019 LA) results in a massive spike in false alarms when applied to compressed audio (2021 DF), even if that audio is authentic.
  • Ranking vs. Classification: While the model remains good at ranking deepfakes (putting fakes at higher scores than real speech) under domain shift, the absolute scores move so much that a fixed decision boundary becomes useless.
  • Adaptation Limits: Unlabeled threshold adaptation works well in-domain even with 10% spoof contamination. However, under the severe domain shift of the 2021 DF set, even an "oracle" threshold (knowing which samples are real) cannot restore high performance because the distributions of real and fake speech begin to overlap significantly (see Figure 4).
Fig. 4. Score distributions on 2021 DF dataset. Overlap between bona fide and spoof scores explains why a source-calibrated threshold transfers poorly.

Key Results

  • In-Domain (2019 LA): MLN-WavLM achieved an 8.45% Equal Error Rate (EER), remarkably close to the 7.86% EER of a fully supervised baseline.
  • Cross-Domain (2021 DF): Performance degraded sharply to 22.12% EER. For comparison, the supervised baseline also degraded but stayed stronger at 14.94% EER.
  • Threshold Shift: Using the 99th percentile threshold from the source domain led to a 20.23% false positive rate on the 2021 DF set—an unacceptable level for most real-world applications.
  • Adaptation Success: Robust Median/MAD adaptation successfully maintained a 1% false-alarm budget in-domain even when the stream was contaminated with up to 10% deepfakes (see Figure 5).
2019 LA dataset: unlabeled threshold adaptation versus stream contamination. Robust MAD transfer keeps recall up to about 10% spoof prevalence.

Limitations and Future Work

  • Computational Cost: Even though the model isn't "trained," running inference on a large model like WavLM is computationally expensive.
  • Simple Pooling: The use of temporal mean pooling might miss short-duration artifacts or local manipulations.
  • Gaussian Assumption: The Mahalanobis distance assumes a unimodal Gaussian distribution, which may be an oversimplification for the vast diversity of human speech.
  • Future Direction: The authors suggest that future work should focus on updating the normality model itself (the bona fide distribution) in an unsupervised way during deployment, rather than just adjusting the decision threshold.

Summary updated

Abstract

Audio deepfake detectors trained with known synthesis attacks can learn generator-specific shortcuts that fail as synthesis systems and acoustic conditions evolve. We investigate an attack-free alternative: a normality model trained exclusively on bona fide speech, which scores synthetic audio by its deviation from authentic representations. We propose MLN-WavLM, a method that applies shrinkage-Mahalanobis modeling to pooled hidden states from a frozen large-scale, pre-trained speech processing WavLM Base+ architecture. Evaluated on ASVspoof 2019 LA, our best attack-free variant achieves an 8.45\% Equal Error Rate (EER), closely tracking a fully supervised WavLM-HM-Conformer baseline (7.86%). However, when subjected to the ASVspoof 2021 DF domain shift, performance drops to 22.12% EER, exposing vulnerabilities in the bona-fide-calibrated operating point. Layer-wise analysis reveals that early WavLM representations dominate the ensemble. To test real-world viability, we simulated deployment streams using unlabeled threshold adaptation. While robust median/MAD recalibration easily handles in-domain contamination up to 10%, recalibrating under the 2021 DF domain shift triggers a total collapse of low-FPR recall. We conclude that while frozen source-domain normality can effectively rank deepfakes, it does not yield a stable open-world decision rule, and downstream threshold adaptation cannot rescue upstream ranking degradation.
BibTeX
@misc{niekurzak2026when,
  author = {Daniel Niekurzak and Daniel Kozuch and Weronika Folwarska and Khan Muhammad and Rafał Scherer},
  title = {{When Normality Shifts: Multi-Level WavLM Modeling for Open-World Audio Deepfake Detection}},
  year = {2026},
  eprint = {2611.11465},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11465},
}
Figure