Self-Familiarity Blind Spots in Audio Deepfake Detectors

Audio deepfake detectors built on large pretrained speech encoders generalize better than older spectrogram classifiers, but they still fail unpredictably on new generators. This note proposes one specific and testable cause. Modern speech generators are increasingly built from the same parts that detectors use. A voice clone may be decoded from HuBERT units or from neural-codec tokens such as EnCodec or DAC, and the detector that judges it may run on a HuBERT-family or codec-derived encoder. In text evaluation, a language-model judge favors outputs that look statistically "natural" under its own token probabilities. We ask whether a detector likewise waves through fakes that its own encoder finds unusually easy to reconstruct. If this blind spot exists, part of the well-known generalization gap is not random. It is predictable from the shared lineage between a generator and a detector, and it can be partly corrected. A committee of detectors with different lineages, which discounts each member's vote on clips that its own encoder finds suspiciously familiar, should be harder to fool.

The idea borrows a mechanism from LLM-as-a-judge research, uses multi-agent consensus as the remedy, and treats benchmark contamination as a confound that the evaluation must remove. The application regime is out-of-distribution audio deepfake detection, where models trained on ASVspoof 2019 have been reported to degrade by 200% to 1000% in Equal Error Rate (EER, the operating point where false accepts equal false rejects) on in-the-wild audio (Does Audio Deepfake Detection Generalize?). They also fall to 41.583% EER on codec-based fakes (Codecfake).

Core Claim

The claim has two parts. The first is diagnostic. For a detector $d$ with pretrained backbone $B_d$, the miss rate on a fake clip $x$ rises as $x$ becomes more "familiar" to $B_d$. Familiarity is defined as the backbone's own self-supervised loss on the clip, z-scored against that backbone's loss distribution on real speech:

$$f_d(x) = \frac{\mathcal{L}_{B_d}(x) - \mu_d^{\text{real}}}{\sigma_d^{\text{real}}},$$

Here $\mathcal{L}_{B_d}(x)$ is the masked-prediction loss for an SSL encoder, or the reconstruction error for a codec encoder. The terms $\mu_d^{\text{real}}$ and $\sigma_d^{\text{real}}$ are its mean and standard deviation on held-out bona fide speech. Fakes produced by a generator that shares components with $B_d$ should sit at strongly negative $f_d$ and be missed more often. This is a direct transfer of the perplexity-trap finding of Self-Preference Bias in LLM-as-a-Judge, which shows that self-preference in text judges tracks low perplexity under the judge rather than explicit self-recognition. It also parallels Preference Leakage, which shows judges inflate scores for outputs of related models, up to 37.1% Preference Leakage Score on Arena-Hard, while being unable to tell which outputs came from relatives.

The second part is corrective. A committee of detectors whose backbones come from disjoint lineages can be combined with clip-specific weights:

$$s(x) = \sum_d w_d(x)\, s_d(x), \qquad w_d(x) \propto \exp\big(\lambda \min(f_d(x), 0)\big)\cdot \exp(\lambda_0\, \mathrm{rel}_d).$$

In this rule, $s_d(x)$ is member $d$'s calibrated fake score and $\mathrm{rel}_d$ is its held-out reliability. The weights are normalized to sum to one, and the two strength terms $\lambda, \lambda_0 \ge 0$ are tuned on a development split. Members are down-weighted exactly where they are most likely to be fooled.

What would be new is the lineage-stratified diagnosis of detector failure and a consensus rule derived from it. Neither more forensic agents nor a bigger judge is the contribution.

The hypothesis could also come out with the opposite sign: a shared backbone might make a generator's artifacts more visible. The diagnostic is informative either way. The correction is only claimed if the sign is as predicted.

Relation to Prior Work

The closest cross-direction work is Multi-Agent Forensic Reasoning for Generalizable Deepfake Video Detection. It splits video forensics across texture, lighting, motion and physics agents and combines them with a trained judge, reaching 53.28% F1 on 20 unseen generators against 47.45% for the best single model. Its own ablation constrains our design. A text-only judge that never sees frames still reaches 51.01%, so most of the gain comes from having several views, not from deliberation. We therefore do not port that architecture to audio. We ask why particular views fail on particular generators, and we weight views accordingly.

Committee aggregation methods for text judges are PiCO and PGED. PiCO weights peer reviewers by consistency and reports 0.90 Spearman correlation with Chatbot Arena versus 0.76 for majority voting. PGED removes preference cycles from a graph of weak judges. Both supply aggregation ideas, and PiCO-style reliability weighting appears in our $\mathrm{rel}_d$ term. PiCO also names our central risk: consensus converges on shared bias when members share pretraining. Our weighting is clip-specific and tied to a measurable cause, whereas both methods use global per-judge weights. The cycle-removal machinery of PGED is not needed here, because detectors emit scores, not pairwise preferences.

On the detection side, three results supply our committee members and set the strongest numbers:

DFALLM's finding that semantic encoders discard forensic detail motivates measuring familiarity at the encoder level rather than at the language-model output. ALLM4ADD shows off-the-shelf audio LLMs fail zero-shot and need fine-tuning. That rules out a frontier zero-shot judge as the main competitor.

Contamination enters as a constraint. ASVspoof and In-the-Wild clips are public, and backbones trained on web-scale audio may have seen them. DyVal showed that freshly generated instances expose memorization that static test sets hide. We adopt its principle of generating test items on the fly from a controlled structure, applied here to audio synthesis pipelines.

Research Program

Generator–detector lineage map. The first step builds this map. On the generator side we use:

  • the ASVspoof 2019 LA vocoder systems,
  • the Codecfake codec resynthesis systems F01–F07, including its unseen DAC split,
  • new resynthesis pipelines we construct, which decode speech from k-means units of HuBERT, wav2vec 2.0 and XLS-R, and from EnCodec, SoundStream and DAC tokens.

On the detector side we train identical AASIST-style back-ends on frozen front-ends drawn from the same families, plus a front-end built on a codec encoder. Every generator–detector pair is labeled as same-encoder, same-family, or disjoint. This gives a factorial design: the same text and speakers pass through generators whose only difference is the component shared, or not shared, with each detector.

Freshness control. The second step adds a DyVal-style generator for test clips. Each test item is assembled at evaluation time from a structure of text, held-out speaker, synthesis component and channel degradation (compression, noise). It is drawn with fresh seeds, so no clip can have appeared in any backbone's pretraining. We report every detector on both static clips and freshly generated clips with matched factors. The static-versus-fresh difference is a contamination indicator in its own right. It also keeps the lineage measurement from being confounded by memorized test audio.

Familiarity analysis. The third step measures $f_d$ for every clip and detector. It tests whether familiarity predicts misses beyond what lineage labels alone explain. This matters because familiarity is computable on any clip at test time, while lineage is often unknown for black-box generators.

Lineage-diverse committee. The fourth step builds the committee and its weighting rule. Committee membership is chosen to maximize lineage disjointness, under the same total parameter and inference budget as the strongest single detector.

Audio LLM explanations. A smaller fifth step reuses the cue-acknowledgment test of The Silent Judge, which found a 0% acknowledgment rate for shortcut cues in text judges. We fine-tune a DFALLM-style audio LLM with explanation outputs and check whether its explanations change when its verdict flips on high-familiarity fakes. If the explanation stays the same while the verdict flips, that indicates unfaithful forensic rationales.

Baselines

Naive reference. A single wav2vec2-AASIST detector trained on ASVspoof 2019 LA only. On Codecfake the reported number for an ASVspoof-trained model is 41.583% average EER (Codecfake). On In-the-Wild the best raw-audio model in Does Audio Deepfake Detection Generalize? reached 33.9% EER. This anchors how bad the gap is without intervention.

Strongest single detectors. First, W2V-ASDG. It is the strongest reported single SSL detector on cross-domain transfer, at 4.60% average EER across its cross-domain and cross-language scenarios and 2.22% on ASVspoof2021DF (W2V-ASDG). Second, W2V2-AASIST trained on Codecfake, which reports 0.884% EER on the unseen DAC split (Codecfake). For an audio LLM detector we use DFALLM, which reports 94.07% accuracy on In-the-Wild (DFALLM). The ALLM4ADD figure of 26.99% EER on In-the-Wild is a weaker LLM reference (ALLM4ADD). No reported result was found on Lacuna for any of these on lineage-matched resynthesis or on freshly generated clips. We will retrain each from released recipes on identical training data (ASVspoof 2019 LA plus the Codecfake training split) and report it on every split.

Strongest consensus competitor. Equal-weight score averaging over the same lineage-diverse committee. The PiCO-style variant uses only the global reliability term ($\lambda=0$). Majority voting is reported for completeness. All committees share one total inference budget, matched to the largest single detector by restricting members or distilling. Calibration and the weight strengths get the same development-split tuning budget as single-model thresholds.

Ablations isolating the key ingredient. The first applies familiarity weighting to a lineage-homogeneous committee: several wav2vec-family detectors trained with different seeds. It separates the effect of discounting familiar clips from the effect of lineage diversity. The second replaces familiarity with a generic confidence or out-of-distribution score from each member. This tests whether the specific self-loss signal matters.

Success criterion. Beating the baselines means two things on the same-lineage strata of the fresh split. The familiarity-weighted lineage-diverse committee must reduce EER relative to equal-weight averaging of that same committee, with bootstrap confidence intervals over speakers that exclude zero. It must also not lose more than a small tolerance on disjoint strata and on In-the-Wild.

Why not just…?

Why not just average a diverse ensemble? This is the real threat. Diversity alone may account for most of the benefit, as the 51.01% text-only judge result in the video multi-agent work hints. The decisive comparison is the familiarity-weighted committee versus equal-weight averaging of the identical members. If the gap is negligible, we will report that the contribution reduces to two things: the lineage diagnosis and a committee-selection rule (pick disjoint backbones). That is still actionable, but we would not claim the weighting.

Why not just train on more generator types, including codec fakes? Codecfake shows codec-trained detectors transfer to an unseen codec at 0.884% EER, so data coverage clearly helps within a family. Our question is different. It concerns generators that share the detector's own backbone, which coverage cannot anticipate when new generators reuse new encoders. Every detector in our study is also trained with Codecfake data, so any remaining same-lineage gap is measured after this fix.

Why not just use a frontier multimodal model as a zero-shot judge? ALLM4ADD reports that off-the-shelf audio LLMs fail zero-shot detection. In video, Gemini-3.5-Flash trailed a committee of small specialized models (46.22% vs 53.28% F1). We include one zero-shot audio LLM as a reference on the fresh split but do not expect it to compete.

Why not just hold out generators, as existing benchmarks do? Leave-one-generator-out splits conflate many differences at once. Our factorial resynthesis keeps text, speaker and channel fixed and changes only the shared component. That is what lets the gap be attributed to lineage rather than to generator quality.

Proposed Experiments

The first decisive check is a regression of per-clip miss probability on lineage relation and on $f_d$, fit across all detector–generator pairs on fresh clips, with speaker and text as random effects. The hypothesis survives only if two conditions hold. Same-encoder pairs must show significantly higher miss rates than disjoint pairs at matched generator quality, measured by a speaker-similarity and naturalness proxy. And $f_d$ must remain predictive within lineage strata.

The second check compares static and fresh EER for each detector. A large static advantage, especially for detectors whose backbones plausibly saw ASVspoof or In-the-Wild audio, would show contamination inflating current leaderboards, independently of the lineage result.

The third check is the committee comparison described under Baselines. We report EER on four test sets: In-the-Wild, ASVspoof2021DF, the Codecfake unseen split, and the fresh lineage-stratified split. Each result is broken down by stratum, together with the two ablations.

The fourth check is a black-box transfer test. We hold out one resynthesis family entirely, for example DAC tokens, from both training and lineage labeling. We then ask whether $f_d$ alone, computed with no knowledge of the generator, recovers its blind spots. This is the condition that matters in deployment, because an attacker's generator lineage is unknown.