Detectors for audio-visual deepfakes increasingly work by watching for moments where speech "moves wrong" over time: the sound and the lips drift apart, or the audio features jump in ways real speech supposedly does not. Phonetics research shows that real speakers do not hold their vocal tract still while their pitch changes. When pitch rises, speakers with high voices relative to their vowel resonances, mostly women, shift their first vowel resonance upward to keep vowels intelligible. Men barely do this. This proposal asks a concrete question. Do temporal inconsistency detectors raise false alarms on genuine speech exactly where this natural pitch-driven resonance shift happens? If they do, can a small, speaker-relative model of the expected shift remove those false alarms without erasing gender information?
The core idea fits in two sentences. The rule that resonance shifts with pitch is physical, not categorical: the size of the shift depends on how a speaker's pitch compares to their own first resonance, so it can be estimated for each speaker from their own speech without any gender label. We audit whether leading detectors are biased along this acoustic axis. We then test whether scoring each window against the speaker's own expected pitch–resonance coupling both cuts false alarms and, where the coupling breaks at a spliced-in synthetic word, sharpens forgery localization.
Core Claim
Two published measurements motivate the claim. On-line formant shifting as a function of F0 elicits Czech vowels at low, normal and high pitch. It finds that female speakers raise the first resonance (F1) by a factor of 1.125 across a 1.37 pitch ratio, recovering about 64% of the vowel information lost when widely spaced voice harmonics undersample the resonance. Male F1 stays close to unchanged. From those two numbers, the implied female elasticity is roughly $\log 1.125 / \log 1.37 \approx 0.37$, and male elasticity is near zero. EMALG shows that under noise, men raise pitch 35% more than women, while women raise F1 101% more and vocal brightness (alpha ratio) 81% more. Real speech therefore contains systematic, speaker-dependent co-movement of pitch and resonance within an utterance.
The detectors we target were never told this. Audio-Visual Deepfake Detection With Local Temporal Inconsistencies scores raw distances between audio and video features over short windows, and its authors already note that fine temporal resolution flags natural variation such as compression stutters. AuViRe reconstructs one modality's speech representation from the other and treats reconstruction error as evidence of forgery. Neither conditions on the speaker's articulatory habits. The only gender-aware detection work in the file, FairAlign, is image-only and treats gender as a nuisance attribute to scrub from face embeddings.
Our hypothesis has three parts. First, on genuine clips, detector false alarms rise with within-window pitch excursion. Second, that rise is steeper for speakers with a high median F0-to-F1 ratio. Third, the excess is explained by resonance movement that matches the speaker's expected coupling. This hypothesis is testable and could fail: the visual mouth-opening that accompanies F1 raising might keep cross-modal detectors consistent. We therefore make the audit the first and decisive step.
Research Program
The first component is an acoustic audit that does not depend on labels. For every genuine clip, we track pitch with CREPE and F1/F2 with Burg linear prediction, following the measurement conventions in On the Production and Perception of a Single Speaker's Gender. We then compute each speaker's median ratio $\rho_s = \bar F_{0,s} / \bar F_{1,s}$ and bin windows by pitch excursion. The primary audit axis is $\rho_s$ rather than a gender label. This matters for two reasons. Nonbinary speakers deliberately decouple resonances from sex (Hope et al.). The undersampling mechanism predicts that coupling tracks $\rho_s$, not category. For fairness reporting we also annotate perceived gender on a subset of real clips, so that group false-positive gaps can be stated in the usual way.
The second component is a speaker-relative coupling model. For window $t$ of speaker $s$, the coupling residual is
$$r_t = \Delta \log F_{1,t} - \beta(\rho_s)\,\Delta \log F_{0,t},$$
where $\Delta$ denotes change across the window and $\beta(\rho_s)$ is the expected F1-per-pitch elasticity for a speaker with pitch-to-resonance ratio $\rho_s$. We fit $\beta(\cdot)$ as a monotone function. It is anchored on the Czech controlled data, where pitch is manipulated and every token is real. We refine it on EMALG with an alpha-ratio term added for effort-driven changes. In partial-fake benchmarks the rest of the clip is genuine, so the speaker's own $\rho_s$ and a per-speaker correction to $\beta$ can be estimated from the unmanipulated context. This follows the same logic by which AuViRe exploits within-clip consistency.
The third component uses the model in two directions. Explaining away is about false alarms. We regress a frozen detector's window score on $|\beta(\rho_s)\,\Delta\log F_{0,t}|$ using genuine training clips, and we score test windows by the residual. This removes the part of the detector's alarm that the physics predicts. Detection is about forgeries. We add $|r_t|$ as one extra input to the localization head. A voice-cloned word inserted by SV2TTS, as in Glitch in the matrix, has no reason to reproduce the target speaker's own elasticity, so $|r_t|$ should spike at the splice. Formant tracking on compressed in-the-wild audio is unreliable. We therefore gate both uses on a formant-confidence score and report coverage, which is the fraction of windows where the gate allows the model to act.
Baselines
The naive reference is each detector used unmodified with a single global threshold. It fixes the size of any disparity and the starting localization accuracy.
The strongest localization competitor is AuViRe. It reports a +8.9 [email protected] improvement over prior methods and a video-level AUC of 99.94 on LAV-DF, with +14.2 [email protected] on AV-Deepfake1M. It is the strongest because it already models cross-modal speech consistency at the segment level. We also run BA-TFD+ from Glitch in the matrix through its released code. On LAV-DF it reports [email protected] of 96.30, [email protected] of 84.96 and [email protected] of 4.44.
For video-level cross-modal detection, we use Local Temporal Inconsistencies. It reports 98.0% AUC on DFDC and 87.0% cross-dataset AUC on FakeAVCeleb. It is the detector whose raw window distances most directly invite the false alarms we hypothesize. We also include AV-Lip-Sync+, which reports 99.29% accuracy on FakeAVCeleb.
For fairness, we port FairAlign to the audio-visual embeddings. It reports a 0.16% gender false-positive-rate gap on FF++, but only for visual detectors. No reported audio-visual result for it was found on Lacuna.
We also run a per-group threshold baseline, in which thresholds are equalized by annotated gender. It is the cheapest possible fairness fix.
Fairness of comparison is enforced as follows. All methods share train/test splits and the same frozen backbones. Every added module, including our coupling term and the ablations, gets the same hyperparameter search budget, and thresholds are tuned on the same held-out genuine data.
Ablations isolate the key ingredient in three ways. The first appends raw F0 and F1 trajectories to the detector without the coupling model. The second replaces $\beta(\rho_s)$ with a two-value lookup keyed on the gender label. The third uses a population-average $\beta$ with no speaker adaptation.
"Beating" a baseline has three conditions. At matched overall true-positive rate, our method must reduce the false-positive gap across $\rho_s$ tertiles, and across annotated gender, by more than per-group thresholding and FairAlign. It must do this without lowering LAV-DF [email protected] below the host detector's value. The coupling residual must add localization AP at strict IoU (0.9 or 0.95) over AuViRe, where the reported numbers leave the most headroom.
Why not just…?
Why not just trust AV-HuBERT-style representations to have learned the coupling already? They may have, and that would make the audit null. The file gives indirect reasons for doubt. HuBERT-based regressors lose accuracy specifically on resonance and vocal weight (Towards an Interpretable Representation of Speaker Identity via Perceptual Voice Qualities). AuViRe falls to 75.5 AUC on real-world fact-checked videos with varied speech. The audit settles this directly. If false alarms on genuine clips do not rise with pitch excursion more steeply for high-$\rho_s$ speakers, we report a clean negative and drop the explain-away component.
Why not just remove gender from the representation, as FairAlign does? Invariance training can equalize average error rates, but it discards the variable that predicts when a resonance change is legitimate. The detector is left unable to tell a real pitch-driven F1 rise from a splice. We test this by comparing localization AP and within-group false alarms, not only group gaps. If ported FairAlign matches us on both, the coupling model is unnecessary.
Why not just use per-group thresholds? This is the most serious threat to the fairness half of the proposal. At the video level it can erase an average gap with no modeling at all. It has three limitations. It needs gender labels, which fit nonbinary speakers poorly. It shifts every window's threshold instead of only the high-excursion windows where we predict the errors cluster. It does nothing for localization. We report it at matched true-positive rate. If it closes the gap as well as our method and our residual adds no localization AP, we will say the fairness contribution does not survive.
Why not just feed F0 and formant tracks into the detector? A learned head could in principle discover the coupling. The first ablation measures exactly that. We expect it to underperform where fakes are rare and speaker-specific elasticity must be estimated from little data, because the coupling model supplies that structure as a prior rather than asking the head to learn it.
Proposed Experiments
The first check is the audit. On genuine clips from LAV-DF, FakeAVCeleb real-audio/real-video, and DFDC, we plot segment-level false alarms per minute against pitch excursion, split by $\rho_s$ tertile and by annotated gender, for all four host detectors. We then pass the controlled real speech of the Czech and EMALG corpora through each detector's audio branch. Every alarm there is false, and pitch and noise effort are experimentally controlled. The prediction is that high-pitch and loud conditions raise alarms more for high-$\rho_s$ speakers.
The second check is the explain-away experiment. We report per-tertile and per-gender false-positive rates at a fixed true-positive rate, alongside overall AUC. The criterion is a smaller gap with unchanged AUC.
The third check is the localization experiment on LAV-DF and AV-Deepfake1M. We add $|r_t|$ to AuViRe and BA-TFD+ and report [email protected], 0.75, 0.9 and 0.95, and AR@100. We also run a diagnostic that checks whether gains concentrate on inserted words with large pitch movement, which the mechanism predicts.
The last check is transfer to the 371-clip real-world fact-checking set used by AuViRe. There the non-English speech and variable recording conditions will stress both formant tracking and the assumption that $\beta(\rho_s)$ generalizes across languages. We report gate coverage alongside AUC so a reader can see how often the model had a say.
References
- On-line formant shifting as a function of F0
- EMALG: An Enhanced Mandarin Lombard Grid Corpus with Meaningful Sentences
- Audio-Visual Deepfake Detection With Local Temporal Inconsistencies
- AuViRe: Audio-visual Speech Representation Reconstruction for Deepfake Temporal Localization
- Kernel Space Conditional Distribution Alignment for Improving Group Fairness in Deepfake Detection
- On the Production and Perception of a Single Speaker's Gender
- Nonbinary American English speakers encode gender in vowel acoustics
- Glitch in the matrix: A large scale benchmark for content driven audio-visual forgery detection and localization
- AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos
- Towards an Interpretable Representation of Speaker Identity via Perceptual Voice Qualities