Systems that separate one person's voice from a crowded room work much better when they can watch that person's lips. The lips give a timing signal that the competing voices do not share. They fail badly, though, when the mouth is hidden. A face mask, a microphone, or a hand over the mouth does not leave them at audio-only quality. They do worse than that: they mute the target, add artifacts, or lock onto a different visible talker. The best current repair methods fill in the missing mouth using two ingredients: a clean frame of the same speaker's mouth taken from elsewhere in the video, and the audio itself. That is exactly what is missing in the hardest cases. A speaker wearing a surgical mask for the whole clip offers no clean frame. The audio is the noisy mixture we are trying to clean, so audio-guided lip synthesis risks inventing mouth movements that match the wrong voice.
This note proposes to replace that repair step with a cross-modal translation that uses neither ingredient. The translation is a learned mapping from one signal type to another, in the spirit of medical "virtual scanners" that predict a CT scan from an MRI. We train a model to predict, frame by frame, the features a lip-reading encoder would have produced for the mouth, using only the motion of the visible upper face (jaw line above the mask, cheeks, eyes, brows) together with a static identity cue. The prediction happens in feature space, not pixels. It is supervised on ordinary unoccluded talking-face video, where the true mouth features are free labels. The predictor never sees the audio mixture, so its output cannot be pulled toward an interfering talker. It needs no reference frame, so it applies when the mouth is hidden for the entire clip. The claim is that this explicit, densely supervised translation recovers more of the lost visual benefit under clip-long occlusion than three alternatives: static identity anchors, audio-guided lip inpainting, or simply training a separator end-to-end on masked full faces.
Where This Sits
Three lines of prior work motivate the gap. My Lips Are Concealed showed that mouth-centric enhancement collapses under obstruction. Its fix combines memory across time, a voiceprint, and a two-pass "self-enrollment" that bootstraps the voiceprint from unoccluded segments. It keeps 12.2 dB SDR at 80% frame occlusion, but by construction it cannot initialize when no clean segment exists. Restoring Speaking Lips from Occlusion (AVLR) is the closest existing work. It detects occluded frames, synthesizes the lower face from audio plus an unoccluded reference frame, and blends it back. On LRS3 at −5 dB SNR it brings AV-HuBERT's WER from 13.87% under occlusion to 11.87%, against 10.62% clean. Our proposal differs on both of its dependencies: we use no reference frame and no audio conditioning, and we never render pixels. Robust AVSE: Correcting Misassignments documents the failure our design targets. When target visuals degrade, cross-attention latches onto background talkers.
Two results supply the core mechanism and also constrain it. Employing the Complete Face in AVSR found that full-face optical flow recovered nearly 70% of the visual speech benefit under total mouth occlusion. That study used a single speaker saying isolated digits, so scale is untested. Deep AVSS Based on Facial Motion found that facial motion gave 1.7 dB SDRi under partial visuals, against 0.2 dB for raw images. Both tell us the upper face carries timing information. Neither learns to turn that motion into lip-equivalent features, and neither handles missing motion.
The identity literature constrains the design in a way that matters. VisualVoice (VisualVoice, 8.53 dB SDR on VoxCeleb2 under unreliable lip motion) and FaceFilter (2.528 dB SDRi vs. 0.690 dB audio-only; about 80% speaker assignment) show that a static face anchors who is speaking but not when. FaceFilter also degrades on same-gender pairs. Disentangling Identities for Cross-Modal Matching reports that face-voice matching leans mainly on the lower face, which is exactly what a mask removes. We therefore expect a masked-face identity cue to be weak. We treat identity as a secondary conditioning signal, not the main source of gain, and test how much it actually adds.
From the time-alignment literature we reuse methods rather than contributions. AV-HuBERT supplies a frozen visual encoder whose features serve as prediction targets, along with its practice of masking whole modality segments during training. Missingness-resilient Video-enhanced Disfluency Detection supplies the strided-convolution resampling we use to bring 25 Hz video features onto the separator's audio frame rate. From image translation (Anatomically constrained CT translation, comparative synthetic CT study) we take one warning and act on it: reconstruction similarity scores miss task-relevant errors. We therefore judge the translator only by what it does for separation and recognition.
Core Claim
The mouth's speech features are largely predictable from the visible upper face over a short time window. This prediction can be learned from unlimited unoccluded video. Separation and recognition should then be judged by whether they improve, not by how good the synthesized mouth looks. Let $u_{1:T}$ be upper-face motion features over $T$ video frames, $s$ a static identity embedding taken from the visible face, and $z_t$ the frozen AV-HuBERT visual-encoder feature for the true mouth at frame $t$. The translator $g$ is trained on unoccluded clips with
$$\mathcal{L} = \sum_t \lVert g(u_{t-k:t+k}, s)_t - z_t \rVert^2 + \lambda\, \mathcal{L}_{\text{cluster}},$$
where $k$ is a half-window of frames. $\mathcal{L}_{\text{cluster}}$ is a cross-entropy on AV-HuBERT's discrete cluster targets, which stresses the phonetic content of $z_t$ over its appearance. $\lambda$ weights the two terms. At test time, a per-frame occlusion detector routes each frame. Observed mouth features are used where the mouth is visible, and $g$'s predictions are used where it is not. A learned confidence from $g$ scales how much the separator trusts each predicted frame.
Research Program
The first stage builds the translator. We crop the region above a synthetic mask line from VoxCeleb2 and LRS3 faces and extract dense optical flow plus appearance features. We train $g$ as a small temporal Transformer, varying the window $k$ to measure how much context upper-face motion needs. Because $g$ is causal-optional, we can also report a low-latency variant with $k$ frames of future context only, in response to the buffering concern raised in the alignment literature.
The second stage plugs predicted features into two frozen-or-lightly-tuned downstream systems. One is a VisualVoice-style separator for SDR on VoxCeleb2 mixtures. The other is AV-HuBERT for WER on LRS3 under −5 dB noise, matching AVLR's protocol so our numbers sit beside theirs.
The third stage is analysis. We break down per-phoneme where upper-face prediction succeeds (jaw-driven vowels, likely) and fails (bilabial closures, likely). We also test whether identity conditioning helps beyond motion, and we measure the predicted stream's synchrony with clean audio, using an AV-HuBERT-based sync score in the style of AV-Lip-Sync+.
Baselines
The naive reference points are an audio-only separator and the same audio-visual separator with occluded frames zero-padded. FaceFilter's audio-only reference is 0.690 dB SDRi with 49.9% assignment. Afouras et al. report the masked audio-visual model collapsing to 2.7 dB SDR at 80% occlusion. We rerun both on our clip-long-occlusion splits rather than quoting these numbers, since the occlusion regimes differ.
The strongest identity-based competitor is VisualVoice, retrained with the same masked-face inputs and the same data and steps as our system. Its reported 8.53 dB SDR on VoxCeleb2 under unreliable lip motion is the published anchor. Beating it means a statistically reliable SDR gain on clip-long occlusion, with the gain concentrated on same-gender mixtures where static identity is known to fail.
The strongest repair competitor is AVLR. Its reported result is WER 11.87% (from 13.87% occluded, 10.62% clean) for AV-HuBERT on LRS3 at −5 dB SNR. AVLR cannot run without a reference frame, so we give it two concessions: an oracle clean reference frame taken from a different video of the same speaker, and a reference face generated from the voice following Reconstructing Faces from Voices. Matching AVLR-with-oracle while using no reference counts as a win. Beating it at low SNR, where its audio conditioning should hallucinate, is the stronger result.
We also compare against Afouras et al.'s self-enrollment, trained with their mouth-blackout "training for failure" recipe on our data. Under full occlusion it reduces to its memory and pretrained-voiceprint path.
The key ablation removes the explicit translation objective. The same architecture and the same masked upper-face crops are trained end-to-end only on the separation or recognition loss. Every system gets an equal tuning budget, measured in GPU-hours and hyperparameter trials.
Why not just…?
Why not just train the separator end-to-end on masked full faces? This is the real threat. "Training for failure" in My Lips Are Concealed shows end-to-end masking training helps a great deal. Our bet rests on supervision density. The translation loss supervises every frame with a rich phonetic target, and it can use clips that contain no mixtures at all. The end-to-end separation loss gives only a weak, indirect signal about upper-face timing. The ablation above settles the question. If end-to-end matches us at equal data, the claim fails. We would then report the translator only as a data-efficiency tool, testing it at 10% of the training mixtures, where dense supervision should matter most.
Why not just use a static face as the identity anchor? VisualVoice and FaceFilter show identity alone works for "who," but FaceFilter's same-gender weakness and the motion-versus-static gap (1.7 vs. 0.2 dB SDRi in Deep AVSS Based on Facial Motion) indicate it cannot supply "when." Masks also remove the lower face that Disentangling Identities finds identity matching relies on. The same-gender split of our VoxCeleb2 evaluation is where this shows up.
Why not just inpaint the mouth from audio, as AVLR does, with a voice-generated reference face? Conditioning on the mixture brings back the circular dependence, since bad separation produces wrong lips. A face generated from a voice also carries no articulation timing. We test this directly by sweeping SNR from clean to −10 dB. Our predictor's input does not change with SNR, so its advantage should grow as noise increases. If it does not, the circularity argument is wrong.
Decisive Checks
The evaluation uses VoxCeleb2 two-speaker mixtures for SDR, SDRi, PESQ, STOI and speaker assignment accuracy, and LRS3 under AVLR's −5 dB protocol for WER. The core test condition places a synthetic lower-face mask on every frame of the clip. As a secondary distribution-shift check we use AVRobustBench. We found no reported result on Lacuna for that benchmark. No real-mask speech dataset appears in the file, so we will record a small real-mask test set to check that synthetic-mask gains transfer.
Four outcomes decide the proposal. First, the explicit-translation system must beat the end-to-end masked-face ablation at equal budget. Second, under clip-long occlusion it must beat retrained VisualVoice on SDR, especially on same-gender mixtures. Third, its WER advantage over AVLR-with-voice-generated-reference must widen as SNR falls. Fourth, in a misassignment test with a visible, talking distractor face, assignment accuracy must stay near its unoccluded level rather than collapsing as Robust AVSE observed. Per-phoneme analysis will also tell us honestly where upper-face motion cannot replace the lips, which bounds how far this approach can go.
References
- My Lips Are Concealed: Audio-Visual Speech Enhancement Through Obstructions
- Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition
- Robust Audio-Visual Speech Enhancement: Correcting Misassignments in Complex Environments With Advanced Post-Processing
- Employing The Complete Face in AVSR to Recover from Facial Occlusions
- Deep Audio-Visual Speech Separation Based on Facial Motion
- VisualVoice: Audio-Visual Speech Separation With Cross-Modal Consistency
- FaceFilter: Audio-Visual Speech Separation Using Still Images
- Learnable PINs: Cross-modal Embeddings for Person Identity
- Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction
- Missingness-resilient Video-enhanced Multimodal Disfluency Detection
- Anatomically constrained CT image translation for heterogeneous blood vessel segmentation
- A comprehensive comparative study of generative adversarial network architectures for synthetic computed tomography generation in the abdomen
- AV-Lip-Sync+: Leveraging AV-HuBERT to Exploit Multimodal Inconsistency for Deepfake Detection of Frontal Face Videos
- Face Reconstruction from Voice using Generative Adversarial Networks
- $\texttt{AVROBUSTBENCH}$: Benchmarking the Robustness of Audio-Visual Recognition Models at Test-Time