Audio deepfake detectors look nearly perfect on the benchmark they were trained on and then fail on real-world clips. Detectors trained on the ASVspoof 2019 benchmark see their error rates grow by 200% to 1000% on in-the-wild celebrity and political audio, according to Does Audio Deepfake Detection Generalize?. The usual explanation is shortcut learning. The detector relies on quirks of the particular speech generators it saw, such as a vocoder's noise floor in a narrow frequency band, and those quirks vanish when the generator changes. These detectors are also large, so practitioners want to prune them, which means deleting most of their weights or channels. Today they must choose between a cheap one-shot cut that damages accuracy and the lottery-ticket recipe, which retrains the network many times. This note proposes that the pruning decision should be made by asking which parts of the network carry shortcuts.
The proposal has two parts. First, we score each channel of a spectrogram-based detector (a model that reads audio as a 2D time-frequency image) by how unevenly it helps across the different generators in the training set. We then remove, in a single step, the channels that help a lot on a few generators and little on the rest. If the shortcut account is right, this is both far cheaper than iterative pruning and less prone to overfitting than the dense model. Second, the same score tells us which channels are hard to remove without losing generalization. We place an ownership watermark in exactly those channels, so that a thief who prunes the watermark away also degrades the stolen detector on real-world audio.
Core Claim
In a spectrogram deepfake detector trained on several known generators, a channel's per-generator usefulness is uneven when the channel encodes generator-specific artifacts. One-shot structured pruning that removes high-unevenness channels, followed by a single fine-tune, should match or lower in-the-wild error relative to the dense model at roughly half the compute. Its search cost should be a small fraction of iterative magnitude pruning (IMP). IMP, the standard lottery-ticket procedure from The Lottery Ticket Hypothesis, trains the network, removes the smallest weights, rewinds the survivors, and repeats for many rounds. Its search overhead is the central complaint in Playing the lottery with rewards and multiple languages and Linear Mode Connectivity and the Lottery Ticket Hypothesis.
The secondary claim concerns ownership. A black-box watermark anchored in the low-unevenness channels should make watermark removal and generalization loss go together. Nobody has measured whether pruning a forgery detector widens or narrows its cross-dataset gap. That question is open in its own right, and our experiments answer it for magnitude pruning and IMP as a by-product.
Research Program
Scoring. For each channel $c$ and each training generator $g$ (the attack labels that ship with ASVspoof 2019 Logical Access), we compute an attribution $A_{c,g}$. It is the average, over fake clips from $g$, of the channel's activation times the gradient of the "fake" output with respect to that activation. This is a standard first-order estimate of how much the channel contributes to catching $g$. The channel's score is
$$ s_c = \operatorname{mean}_g A_{c,g} - \lambda \, \operatorname{std}_g A_{c,g}, $$
where $\lambda \ge 0$ trades raw usefulness against unevenness. At $\lambda = 0$ the method reduces to ordinary first-order importance pruning, which gives us a built-in ablation. Channels with the lowest $s_c$ are removed in one step, up to a target fraction of FLOPs, and the network is fine-tuned once. The total cost is one dense training run, one forward-backward pass over a calibration set, and one fine-tune. We will report this cost in GPU-hours next to IMP's multi-round cost.
Architectures. We use two spectrogram detectors. The first is a convolutional detector on log-linear and constant-Q spectrograms. Müller et al. found these inputs gave an average 37% lower error than mel spectrograms, because mel filtering compresses the high frequencies where generation artifacts live. The second is the transformer AST: Audio Spectrogram Transformer, which splits the spectrogram into patches like an image. For the transformer, the pruned units are attention heads and hidden units of the feed-forward layers. Long patch sequences make this model expensive, so it is where compression matters most.
Watermark. The watermark is a secret set of trigger clips. Each is bona fide speech overlaid with a faint, fixed time-frequency pattern, and each should be classified as "fake." The direction matters for a forensic tool. A trigger that turned fakes into "real" would hand attackers a free evasion key, so the watermark only ever pushes toward "fake." We train the trigger behavior with an added penalty that forces the trigger response to flow through the channels kept by the score. We also apply the adversarial weight perturbation of Watermarking PLMs on Classification Tasks by Combining Contrastive Learning with Weight Perturbation. That paper pushes the watermark into a wide, flat region of the loss surface so that it survives removal of 50% to 70% of parameters in text classifiers. We reuse that technique and change where the watermark lives.
Baselines
The naive reference point is the dense detector, trained on ASVspoof 2019 LA with the same front-end and evaluated on the in-the-wild set. The closest published reference is Müller et al., who report raw-audio models reaching 33.9% EER (equal error rate, where false accepts equal false rejects) on that set. We will retrain our own dense models rather than borrow numbers, so that every comparison shares data and fine-tuning schedule.
The pruning competitors all run at matched FLOPs, from the same dense checkpoint, with the same fine-tuning budget and tuning grid. The first is one-shot magnitude pruning with fine-tuning. The second is the first-order importance score with $\lambda = 0$. The third is IMP with rewinding, which is the strongest competitor on accuracy because it is the method whose subnetworks are known to match dense performance. IMP gets its full multi-round budget, and we report how much compute that is. We found no reported result on Lacuna for any pruned audio deepfake detector on the in-the-wild set, so the bar is set by our own runs.
The method beats the competitors only if two things hold at 50% FLOPs. Its in-the-wild EER must be at or below the dense model and IMP. Its total search cost must be at most a quarter of IMP's.
For broader context, Alethia reports 5.2% average EER across 50 speech deepfake benchmarks using 19,000 hours of pretraining. Audio Deepfake Verification reports 0.03% EER on ASVspoof 2019 LA. We will not claim to beat these large foundation models. They mark the ceiling for the compute-constrained regime we target.
The watermark baselines are a plain backdoor trigger set as in Turning Your Weakness Into a Strength and the flat-minimum watermark of Gu et al. without the channel anchoring. The ablation that isolates our ingredient anchors the watermark in randomly chosen channels instead of score-selected ones. On image classifiers, You are caught stealing my winning lottery ticket! reports a mask signature that stays decodable after an extra 20% of weights are pruned. We cite that as context, not as a directly comparable number.
Why not just…?
Why not just use one-shot magnitude pruning and fine-tune? The research file describes a well-known quality crash after aggressive one-shot pruning. However, the cited evidence comes mostly from diffusion and language models (Pruning for Sparse Diffusion Models based on Gradient Flow), not from deepfake detectors. Magnitude pruning might accidentally discard shortcut channels and help generalization. This is a genuine possibility, and it is exactly why magnitude pruning is a matched baseline. If it equals our score on in-the-wild EER, the shortcut account of pruning is wrong and we will report that.
Why not just train with frequency-band masking or other augmentation? This is the most serious threat. Randomly masking frequency bands during training also discourages reliance on any single band, and it costs nothing at search time. We will train dense and pruned models with and without band-masking augmentation. If augmentation alone closes the gap as much as score-based pruning, the generalization claim collapses. The contribution would then shrink to cheap compression that does not hurt generalization, which is still useful but smaller. If the two effects stack, that is evidence the score removes shortcuts that augmentation does not reach.
Why not just switch to a better front-end or a foundation encoder? Choosing a log-linear or constant-Q spectrogram over mel is already the largest cheap gain reported by Müller et al., so all our models use it. Foundation encoders such as Alethia generalize better, but they cost tens of thousands of hours of pretraining and are hard to deploy. Our question concerns what can be done at a fixed, small budget. A better front-end is complementary to pruning, not a replacement.
Why not just prove ownership by comparing weights, as AWM does? AWM reaches 1.0 AUC at telling related language models apart without any training. It needs the suspect model's weight matrices, however. A stolen detector served behind an API exposes only scores, so black-box triggers remain necessary there. Where white-box access is available, we will run an AWM-style alignment on our pruned and fine-tuned thefts as a complementary check.
Proposed Experiments
We train on ASVspoof 2019 LA. Testing uses three sets: its evaluation split, the 37.9-hour in-the-wild set from Müller et al., and the MLAAD and ADD2023 sets used by Audio Deepfake Verification as further unseen-generator tests. Metrics are EER per set, FLOPs, parameter count, and search GPU-hours. We sweep target sparsity and $\lambda$.
The first decisive check is a leave-one-generator-out test inside ASVspoof. We compute scores without one attack type, then check whether the high-unevenness channels predict the error on the held-out attack. If unevenness does not predict held-out error, the mechanism is not doing what we claim, whatever the headline numbers say.
The second decisive check is the generalization-versus-sparsity curve on the in-the-wild set for every pruning method. It answers the open question of whether pruning widens or narrows the gap.
For ownership, we attack the watermarked, pruned detectors with three removal attacks. These are further magnitude pruning up to 70%, fine-tuning on a small held-out set, and distillation into a fresh student, which mirrors the attacks in Box-Free Model Watermarks Are Prone to Black-Box Removal Attacks. For each attacked model we record trigger accuracy and in-the-wild EER. The anchoring hypothesis passes only if two conditions hold. Attacks that drive trigger accuracy to chance must raise in-the-wild EER substantially more for score-anchored watermarks than for randomly anchored ones. The false-trigger rate on clean bona fide speech must also stay at the level of the unwatermarked model. Distillation is the attack most likely to break this coupling, so we will report it separately.
References
- Does Audio Deepfake Detection Generalize?
- The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
- Playing the lottery with rewards and multiple languages: lottery tickets in RL and NLP
- Linear Mode Connectivity and the Lottery Ticket Hypothesis
- AST: Audio Spectrogram Transformer
- Watermarking PLMs on Classification Tasks by Combining Contrastive Learning with Weight Perturbation
- Alethia: a Foundational Encoder for Voice Deepfakes
- Audio Deepfake Verification
- Turning Your Weakness Into a Strength: Watermarking Deep Neural Networks by Backdooring
- You are caught stealing my winning lottery ticket! Making a lottery ticket claim its ownership
- Pruning for Sparse Diffusion Models Based on Gradient Flow
- AWM: Accurate Weight-Matrix Fingerprint for Large Language Models
- Box-Free Model Watermarks are Prone to Black-Box Removal Attacks