Too Predictable from the Face: Two-Sided Reconstruction Tests for Face-to-Voice Deepfakes

Face-to-voice attacks generate a stranger's voice from a single photo of their face. Current audio deepfake detectors mostly miss them. One detector posts an equal error rate of 0.566 on such speech, worse than chance, and detectors fine-tuned to catch them forget how to catch other fakes (Can Current Detectors Catch Face-to-Voice Deepfake Attacks?). We propose a detector trained only on real people talking. It asks how much seeing a speaker's face helps it fill in masked-out parts of their voice. For a real speaker, the face helps a moderate, characteristic amount, because skull and jaw shape constrain pitch and timbre without fully determining them. A voice dubbed in from another person gets less help than that from the face. A voice manufactured from the face gets more help than any real voice does. The detector scores both too little and too much help as anomalous.

The point of the two-sided test is the weakness of existing cross-modal detectors. Embedding matchers that check whether a voice "fits" a face report that they break down when both sides are made to match the same identity (Voice-Face Homogeneity Tells Deepfake). A face-to-voice attack is exactly that case: the voice fits the face by construction. Our wager is that it fits too well. Synthesizers driven only by a face are known to drift toward the average voice for that kind of face. Real people carry accent, register, and individual quirks that the face cannot predict. That missing unpredictability is what the detector looks for.

Core Claim

We train a generative reconstruction probe on genuine audio-visual speech. A reconstruction probe is a model that learns to repair corrupted normal data, so that large repair errors flag abnormal inputs. Ours is a flow-matching Diffusion Transformer, a generator that learns to turn noise into mel-spectrograms along nearly straight paths. It is trained on bona fide talking-face clips to fill in masked mel-spectrograms. It runs in two modes, with and without a static face embedding as extra conditioning.

Let $x$ be the mel-spectrogram, $m$ a random mask with masking ratio $\rho$, and $f$ the face embedding. Let $\hat{x}_{\varnothing}$ and $\hat{x}_{f}$ be the reconstructions without and with the face. The per-ratio face gain is

$$G_\rho(x,f) = \mathbb{E}_m\big[\,\|(1-m)\odot(x-\hat{x}_{\varnothing})\| - \|(1-m)\odot(x-\hat{x}_{f})\|\,\big],$$

which is the error reduction that the face buys on the masked region.

On genuine pairs, $G_\rho$ follows a distribution $P_{\text{real}}$ that we estimate on held-out real identities. The anomaly score is the two-sided tail probability of an observed gain under $P_{\text{real}}$. We pool this score across ratios $\rho \in \{0.5, 0.75, 0.9\}$, the schedule shown to help in Audio-Anchored Fusion of Multi-Ratio DiT Reconstruction Residuals.

We predict where each kind of input lands. A voice spliced onto someone else's face should give low or negative gain. A face-to-voice output should give abnormally high gain, especially at ratio 0.9, where the remaining audio context says little and the face carries most of the information.

The new element is a reconstruction residual that is differential and cross-modal. Prior reconstruction detectors measure one-sided error within a single modality: audio-only masked spectrograms in the audio-anchored DiT work, masked facial landmarks in Learning Local Reconstruction Errors for Face Forgery Detection, and face style vectors in De-Fake. None of them can detect a voice that is internally clean but derived from a face. Taking the difference between the two modes also cancels much of the sensitivity to recording conditions that plagues raw residuals, because codec and noise effects largely hit both reconstructions alike.

Research Program

The probe borrows from two sources. The masked flow-matching architecture follows the audio-anchored DiT probe, which reconstructs masked spectrograms with a model trained on 2.55 million bona fide utterances. The idea that a static face carries usable vocal information comes from cross-modal identity work. Seeing Voices and Hearing Faces matches voices to faces at 63.9% even after removing gender, age, and nationality cues. Learnable PINs learns joint face-voice embeddings that carry identity detail beyond demographics. VisualVoice uses a static face as a vocal prior when the lips are occluded. Hear Your Face predicts mean pitch from a face to within 24.01 Hz for male speakers, against a dataset standard deviation of 29.18 Hz.

That HYFace result motivates the design and also sets an expectation. The face explains only part of the pitch variance, so real-pair gain should be moderate and spread out rather than near zero or near total. The same literature describes face-only synthesizers collapsing toward average timbres, and this is the mechanism our high-tail hypothesis depends on.

We train on VoxCeleb2 and LRS3 using identity-disjoint splits. During training, the face is dropped with probability 0.5, so a single network learns both modes. We deliberately take the face encoder from a different family than the attack generators use, so we can test whether the effect depends on shared features. ID-FaceVC (Seeing Your Speech Style) uses CLIP, FaceSpeak (FaceSpeak) uses FaRL, and the hierarchical video-to-speech model (From Faces to Voices) uses ArcFace.

Beyond the scalar score, we produce gain maps over time and mel frequency, so we can see which bands carry the excess predictability. We expect the low-frequency bands that encode pitch and formant envelopes to dominate. The audio-anchored DiT work already found residual structure concentrated near mel bin 20.

An optional supervised stage adds the gain features to a frozen WavLM detector through the anchored additive fusion of the audio-anchored DiT paper. In that design the audio branch keeps a fixed weight of 1.0 and the gain branch only adds a gated correction, which avoids the competitive-fusion collapse that paper documents. The main claim, though, is evaluated zero-shot.

Baselines

The naive reference point is the audio-only masked-reconstruction residual from the same probe with the face switched off. This is our reimplementation of the unimodal DiT probing idea. The cited paper reports 13.84% equal error rate on In-The-Wild and 6.54% on ASVspoof 5 for the full fused system. We found no reported result on Lacuna for that system on face-to-voice speech. The baseline uses identical weights, masks, and compute, so any gain our method shows can come only from the face pathway.

The strongest unimodal competitor is AASIST. On the face-to-voice corpus of Can Current Detectors Catch Face-to-Voice Deepfake Attacks? it reports an equal error rate of 0.163 frozen and 0.003 after fine-tuning. The fine-tuned detectors in that paper generalize poorly; Sun et al.'s accuracy on unseen SpeechT5 speech falls from 38.7% to 3.9%. We run AASIST frozen and fine-tuned on the same splits. Beating it means a lower equal error rate than 0.163 zero-shot, together with no loss on held-out generators where the fine-tuned version collapses.

The strongest cross-modal competitor is the voice-face matching detector of Voice-Face Homogeneity Tells Deepfake, which reports 85.13% AUC on DFDC. We retrain it on the same VoxCeleb2 identities and score it both with its usual one-sided mismatch rule and with a two-sided rule that also flags unusually high similarity. The two-sided version is our most important ablation. It asks whether the "too consistent" signal is already present in a cosine similarity, without any reconstruction.

A physical baseline predicts mean pitch from the face, as HYFace does, and scores the two-sided deviation of observed mean pitch. This tests whether one scalar captures the whole effect.

For audio-visual splicing on FakeAVCeleb (FakeAVCeleb), the reference points are two published results. The unsupervised intra- and cross-modal inconsistency detector reports 96.81% average AUC (Unsupervised Multimodal Deepfake Detection). The supervised correlation-distillation method reports 98.8% (Explicit Correlation Learning). We do not expect to beat these on lip-sync forgeries, which our static-face design deliberately ignores. We report them so that our method's complementary value is measured honestly, through late fusion with the unsupervised detector.

All baselines and our method use the same training identities and the same hyperparameter search budget, with thresholds set on the same held-out real data.

Why not just…?

Why not just fine-tune an audio detector on face-to-voice samples? It works in-domain, reaching 0.002–0.003 equal error rate, but the same study shows fine-tuned detectors losing accuracy on unseen generators. Our method never sees attacks, so it cannot overfit to any one generator. We test this directly on generators held out by family.

Why not just threshold voice-face embedding similarity on both sides? This is the real threat. If faked voices already show abnormally high similarity in a contrastive embedding, the reconstruction machinery is unnecessary. We run exactly this ablation. Our reason to expect a gap is that contrastive embeddings are trained to collapse within-identity variation. That is the quirk-level variation whose absence we hypothesize marks synthesis, and reconstruction error keeps it. If the two-sided cosine matches our AUC within noise, we will report that and recommend the simpler detector.

Why not just compare face-predicted pitch to observed pitch? Pitch is the best-understood channel, but the 24.01 Hz HYFace error against a 29.18 Hz spread shows the face predicts mean pitch only loosely. The gain maps will show whether the excess predictability lies in pitch alone or also in formant and timbre bands. If pitch alone matches our method, the physical baseline wins.

Why not just check lip synchrony? Face-to-voice attacks need not involve video at all. Even when they do, modern lip-sync methods defeat synchrony checks (Explicit Correlation Learning). The static-face test targets a separate cue, and we evaluate it on audio clips paired with still photos.

Proposed Experiments

We build an evaluation pool from four sources: the corpus of the face-to-voice study (15,144 samples drawn from VoxCeleb2, AVSpeech, FOICE outputs, and SpeechT5), the FakeAVCeleb fake-audio categories, and new attacks we generate on identity-disjoint LRS3 faces with HYFace, ID-FaceVC, FaceSpeak, DiVISe (DiVISe), and the hierarchical flow-matching video-to-speech model. We report AUC and equal error rate, overall and per generator family.

The decisive check comes first. Face-to-voice outputs must fall in the high-gain tail of $P_{\text{real}}$, and spliced voices in the low tail. If face-to-voice gain sits inside the real-pair band, the central hypothesis is false.

Three confound controls follow. First, we vary whether the attacker's face encoder overlaps with ours, to rule out shared-feature artifacts. Second, we vary how close the attack is to the training data: attacks on identities seen versus unseen by the generator, and attacks from generators trained on VoxCeleb versus other corpora. Third, we vary recording conditions with added noise and codecs. Under these conditions we expect the differential gain to stay stable while the raw audio-only residual drifts, the benign-shift failure mode of reconstruction detectors.

A ratio sweep should show the high-tail signal strongest at ratio 0.9, where the face matters most.

Finally, to confirm the anchored-fusion variant does not trade away generality, we run its audio-only branch on In-The-Wild and ASVspoof 5. Since those datasets have no faces, the gain branch is gated off. The fused system must stay within noise of the audio-only DiT probe's reported 13.84% and 6.54% equal error rates on those datasets.