A recommender trained on click logs learns that popular items are a safe bet, so it recommends them even when a user asks for something niche. Popularity has a direct route to the score that bypasses actual user–item fit. We argue that detectors which ask "were these two fake clips made by the same generator?" have the same problem, with speaker identity in the role of popularity. Two clips in the same voice look like they share a source because the model's embedding encodes timbre, whether or not the same generator produced them. The recommender literature has a cheap, model-agnostic fix for this kind of shortcut: learn the shortcut's contribution separately, then subtract it at test time. We propose to carry that fix into audio deepfake verification and to build the evaluation that shows whether the shortcut exists at all.
This matters because source verification is where synthetic speech forensics is heading. Modern zero-shot generators are close to indistinguishable from human speech, and closed-set attribution breaks on generators it has never seen. Audio Deepfake Verification introduced the open-set verification task and its Audity model, and its authors name speaker–generator disentanglement as the open problem. Their artifact branch is built on CAM++, which is a speaker-verification network. So there is a concrete reason to expect speaker information to leak into the forensic embedding.
Core Claim
Our first claim is that current deepfake verifiers partly score speaker similarity rather than generator similarity. This should be most visible in pairs that cross the two factors: same generator with different speakers, and different generators with the same speaker.
Our second claim is that a counterfactual correction removes much of this confound without retraining the backbone or collecting speaker-matched training pairs. The correction follows the recipe of MACR. MACR adds auxiliary heads that capture popularity's direct effect on the score and subtracts that effect at inference. Here the auxiliary head captures the part of the verification score that speaker similarity alone can explain.
For a clip pair $(a,b)$, let $m(a,b)$ be the verifier's matching score from its fused embeddings. Let $r(a,b)$ be a speaker-only score computed from a frozen speaker-verification embedding. Let $\sigma$ be the logistic function. During training, the same-source prediction is $\hat y = m(a,b)\,\sigma(r(a,b))$, and the speaker head also gets its own auxiliary loss. At inference we score with
$$ s(a,b) = m(a,b)\,\sigma(r(a,b)) - c\,\sigma(r(a,b)), $$
where $c$ is a reference matching value representing "no generator information." The subtracted term is the score the model would give if only the speaker path were active, which is MACR's natural direct effect with speaker in place of popularity.
MACR is used here as a method supplier. What is new is the causal assignment: the speaker plays the role of the confounder, the generator plays the role of the matching path, and the target is pairwise verification rather than ranking.
Research Program
The work has three parts.
The first is a speaker-crossed verification protocol. We take reference speakers from LibriTTS and VCTK and clone each one through several open zero-shot generators, such as VoiceCraft and the editing and synthesis modes of SSR-Speech. Every speaker then appears under every generator, and every generator voices every speaker. Pairs fall into four cells: same or different generator, crossed with same or different speaker. The quantity we care about is the crossed-cell error, which is equal error rate (EER, the operating point where false accepts equal false rejects) computed only on the two conflicting cells.
We also examine ASVspoof 5. It builds 32 attacks over a shared crowdsourced speaker pool, which could supply crossed pairs without new generation. We will check per-attack speaker overlap before relying on it.
The second part is diagnosis. We reimplement Audity's two-branch design, a w2v-BERT 2.0 structural branch plus a CAM++ artifact branch, and train it on MLAAD. We then measure three things. The first is how well speaker similarity alone predicts its same-source score. The second is whether its crossed-cell EER is much worse than its uncrossed EER. The third comes from Audity's reported enrollment effect: its EER falls from 25.0% with one enrollment clip to 4.2% with five on ADD2023 (Audio Deepfake Verification). We test whether that drop comes partly from averaging away speaker identity, by comparing five enrollment clips from one speaker against five from different speakers.
As a secondary diagnostic taken from Adap-τ, we check whether embedding norms grow with how often a generator appears in training. Adap-τ showed that frequent items get longer embeddings, which dominate inner-product scores. If the same happens here, we normalize before scoring. We do not treat this as a main contribution.
The third part is the counterfactual correction above. We apply it to Audity-style models and to a plain single-backbone verifier, and we study how sensitive it is to $c$.
Baselines
The naive reference is the cosine similarity of the trained Audity-style fused embeddings with no correction. Every method is compared against it on identical pairs.
The strongest existing method is Audity itself. Audio Deepfake Verification reports an EER of 10.02% (accuracy 89.87%) on MLAAD and 25.0% one-shot / 4.2% five-shot EER on ADD2023. We first reproduce roughly these figures under the same training data and backbones. Our method then reuses those exact checkpoints, adding only a speaker head and the inference subtraction, so compute and data are matched. Beating Audity means lowering crossed-cell EER without raising the MLAAD and ADD2023 EERs by more than seed-to-seed variation.
The first data-side competitor is speaker-controlled pair mining. It trains on our crossed pairs, forcing the model to separate same-speaker/different-generator pairs from different-speaker/same-generator pairs. It gets the same number of training steps and the same tuning budget as our method. This is the most serious competitor.
The second is adversarial speaker removal: a gradient-reversal speaker classifier on the fused embedding, with its weight tuned over the same grid size as $c$.
The third is linear speaker-subspace projection. We regress the forensic embedding on the frozen speaker embedding and keep only the residual, which is the simplest post hoc fix.
The ablation that isolates the key ingredient keeps the full procedure but feeds the speaker head shuffled speaker embeddings. If the gain survives the shuffle, it comes from the extra head or the extra tuning, not from removing speaker information.
Why not just…?
Why not just mine speaker-controlled hard negatives during training? If matched pairs exist, this may work as well as counterfactual scoring, and we treat that as a real possibility. The counterfactual correction has two claimed advantages. It needs no generator-by-speaker grid at training time, and such grids are hard to obtain for commercial systems. It also applies after the fact to deployed checkpoints. The experiment settles the question. We train the mined-pair model on grids built from open generators, then test both methods on generators and speakers left out of that grid. If mining matches us on held-out generators, the proposal's lasting contribution is the crossed protocol and the diagnosis, and we will say so.
Why not just remove speaker information adversarially? Gradient reversal is known to be unstable, and with CAM++ the speaker and artifact cues may share features. Removing speaker information from the whole embedding could then also strip away generator cues. The counterfactual subtraction touches only the score, not the representation. We compare the two methods on crossed-cell EER and also on uncrossed EER, where over-removal would appear as a loss.
Why not just enroll more clips, or use a larger backbone? Averaging five enrollment clips already cuts ADD2023 EER sharply, and that may already be implicit speaker averaging. The multi-speaker versus single-speaker enrollment test will show whether it is. Larger capacity is unlikely to help: Harder or Different? finds that cross-dataset failures come mostly from shortcuts in the training distribution, and that bigger detectors latch onto them more strongly. In any case, forensic practice often has only one suspect clip, which is the setting our method targets.
Decisive Checks
The first check is whether the confound exists. The proposal stands only if the uncorrected verifier's crossed-cell EER clearly exceeds its uncrossed EER, and speaker similarity alone predicts a meaningful share of its scores. If not, the premise fails and we will report a negative result.
The second check is whether the counterfactual score beats all three competitors on crossed-cell EER for generators held out of training. The shuffled-speaker ablation must recover none of that gain.
The third check targets commercial systems. Audity's authors report that commercial systems such as OpenAI, Azure, and Baidu overlap in embedding space. Our hypothesis is that, once their artifacts are smoothed away, speaker is the main remaining signal. We test whether the correction helps more on commercial-origin pairs in MLAAD than on open-source ones, since that is where it should matter most.
References
- Audio Deepfake Verification
- Model-Agnostic Counterfactual Reasoning for Eliminating Popularity Bias in Recommender System
- VoiceCraft: Zero-Shot Speech Editing and Text-to-Speech in the Wild
- SSR-Speech: Towards Stable, Safe and Robust Zero-shot Text-based Speech Editing and Synthesis
- ASVspoof 5: Design, collection and validation of resources for spoofing, deepfake, and adversarial attack detection using crowdsourced speech
- Adap-τ : Adaptively Modulating Embedding Magnitude for Recommendation
- Harder or Different? Understanding Generalization of Audio Deepfake Detection