Target speaker extraction pulls one person's voice out of a crowded recording, given a cue about who that person is: a short enrollment recording of their voice, or video of their face. These systems are trained as if the target never stops talking. When the target falls silent, the network still has to produce something, and it often produces the wrong thing. It amplifies another talker, or it builds target-like speech out of noise. This failure is called target speaker hallucination. It inserts phantom words into target-speaker transcription and raises false alarms in speaker verification (Towards Robust Speaker Verification with Target Speaker Enhancement). The two standard fixes each cost something. Training the extractor to output silence on target-absent inputs lowers quality when the target is speaking. Re-checking the output's voice against the enrollment works well only with 15–20 second enrollments (Listen only to me! How well can target speech extraction handle false alarms?).
We propose a different check. When the face is available, verify the extracted audio against the target's moving lips, frame by frame, instead of against a voice print. Hallucinated speech, whether it is a leaked interferer or noise reshaped to sound like the target, is not synchronized with the target's mouth. A frame-level audio–visual agreement score on the output is therefore direct evidence of hallucination, and it does not depend on how long the enrollment is. This is where the cross-modal representation gap comes in. Audio and visual embeddings from contrastively trained encoders sit in separate regions of the shared space (Towards understanding the modality gap in CLIP). As a result, raw cross-modal similarity scores are squeezed into a narrow, condition-dependent band. We argue that closing this gap with a simple linear correction is what turns the agreement score into a calibrated presence probability. A calibrated probability can drive a soft gate whose threshold carries over across datasets and noise conditions.
Core Claim
Hallucination in audio-visual extraction can be caught after extraction, without retraining the extractor to be silent. The tool is a small critic that compares the extracted speech with the target's lip video at every frame. Two ingredients make the critic work. First, it is trained on hard negatives drawn from the extractor's own hallucinated outputs on target-absent mixtures. Second, its audio and visual embeddings are re-centered and linearly bridged before scoring, so that the score means the same thing across speakers and conditions. Because the extractor itself is trained only on active speech, the method should avoid the active-speech degradation of null-target training. On sparse-overlap conversational data it should reach equal or lower target-absent output power than the best input-side method. When video drops out, the critic falls back to a gap-corrected identity score against the enrollment, and it does not inpaint visual features from possibly hallucinated audio.
Relation to Prior Work
Four lines of work motivate the proposal by showing the failure and the cost of current fixes. The overview of neural target speech extraction (Neural Target Speech Extraction: An Overview) states the always-active assumption explicitly. Null-target training penalizes output energy on target-absent mixtures (X-TaSNet). Its direct-silence variant TSE-IS drops active-speech SDR improvement from 12.4 dB to 10.8 dB and raises the failure rate from 3.4% to 8.6% (Listen only to me!). TSEJoint adds a presence-detection head inside the extractor and hard-gates the output (Speaker Extraction with Detection of Presence and Absence of Target Speakers). These methods change the extractor, or decide presence once per utterance from audio identity. We leave the extractor's training untouched and decide presence per frame from cross-modal synchrony.
The audio-visual extraction papers are the closest work. USEV conditions on lip video and applies energy losses in target-quiet segments, using ground-truth scenario labels (USEV). ActiveExtract replaces lip-reading features with features from a model pretrained for active speaker detection (deciding from video and audio whether a visible face is currently talking). Its TalkNet features enter the separator as inputs, and it beats USEV by more than 4 dB SI-SNR on IEMOCAP-2Mix (Audio-Visual Active Speaker Extraction for Sparsely Overlapped Multi-talker Speech). Both methods use the visual cue on the input side, to steer the separator. Neither inspects the output. A separator steered correctly can still leak an interferer when babble corrupts the active-speaker features, and nothing downstream checks for it.
Two further papers constrain the design. ImagineNET inpaints missing visual embeddings from the partially separated audio (ImagineNET). During target absence under occlusion, this lets a hallucination justify itself, which is why our fallback never generates visual evidence from audio. Uni-Net avoids cross-modal fusion entirely by running separate audio-cued and video-cued extractors (Unified Audio Visual Cues for Target Speaker Extraction). It was never evaluated on target absence. $C^2$AV-TSE learns frame-level confidence, but only for extraction quality during active speech ($C^2$AV-TSE). Our critic is a frame-level confidence model aimed specifically at presence.
Two papers supply the gap-correction method. Towards understanding the modality gap in CLIP shows that contrastive training itself leaves image and text embeddings perfectly linearly separable. Extending Multi-modal Contrastive Representations finds that the gap inside one pretrained dual encoder is approximately linear: a linear layer bridges it, while a nonlinear MLP distorts the geometry. We reuse that finding directly. One related idea we do not adopt as the main method is MMD-based global distribution alignment (Learning Robust Visual-Semantic Embeddings); it appears only as an ablation.
Research Program
The system has three parts. The first is a standard audio-visual extractor trained only on target-active speech. We use a USEV-style lip-conditioned separator and, separately, ActiveExtract as backbones, so the result does not hinge on one architecture.
The second part is a frame-level critic. It embeds short windows of the extracted waveform, $a_t$, and the target's lip crop, $v_t$, using encoders initialized from TalkNet's audio and visual branches. It then scores agreement as
$$ s_t = \cos\big(W(a_t - \mu_a),\; v_t - \mu_v\big), $$
where $\mu_a$ and $\mu_v$ are the mean audio and visual embeddings over target-present training frames, and $W$ is a single linear map. Following the Ex-MCR finding, we deliberately keep $W$ linear. A logistic layer turns $s_t$ into a presence probability $p_t$. The critic is trained contrastively with three kinds of negatives:
- the extractor's own outputs on target-absent mixtures, which are real hallucinations;
- interferer speech paired with the target's lips;
- target speech shifted in time against the lips.
The third part is the gate. At test time the output is scaled by a smoothed $p_t$, a soft gate rather than a zero/one switch. Sweeping the logistic offset traces the full trade-off between false alarms and missed speech. That is exactly the test-time control TSE-IS was reported to lack.
When lip frames are occluded, the critic switches to scoring the extracted audio against the enrollment voice embedding. This identity score is gap-corrected in the same way. We keep the switch-over explicit and never inpaint visual features from audio.
The analysis component asks whether the modality gap actually matters for presence detection. We measure the linear separability of audio and visual critic embeddings before and after correction. We then compare the score distributions for present and absent frames. The key test is threshold transfer: a threshold fit on VoxCeleb2-mix is applied unchanged to IEMOCAP-2Mix. A constant offset in embedding space need not change the ranking of frames within a single condition. We therefore expect gap correction to matter most for calibration across conditions, and less for threshold-free detection accuracy. We will report both, and a null result on transfer would be a finding in its own right.
Baselines
The naive reference point is the same extractor trained on active speech only, with no gate. Its target-absent output power shows how much hallucination there is to remove. Listen only to me! reports that such a baseline barely attenuates inactive-speaker inputs on LibriMix.
The strongest existing audio-visual method is ActiveExtract. It is the right competitor because it already uses active speaker detection features and is reported to beat USEV by more than 4 dB SI-SNR on IEMOCAP-2Mix. For target-absent output power, the file reports only a "significant reduction" for ActiveExtract and gives no scalar, so no reported result was found on Lacuna for ActiveExtract's target-absent output power. We will retrain it on our exact splits. The second strong competitor is USEV with its scenario-aware energy losses, which represents null-target training in the audio-visual setting.
The strongest identity-based gate is a TSE-V-style system. Its reported figures on LibriMix-100 are a presence-detection equal error rate of 8.9% and an SDR improvement of 11.9 dB after detection (Listen only to me!). We run it with the same short enrollments we give our fallback. We also include a TSEJoint-style internal detection head, reported at 10.6% equal error rate on WSJ0-2mix-extr (Speaker Extraction with Detection of Presence and Absence of Target Speakers). For the occlusion conditions we add ImagineNET.
The most important baseline is off-the-shelf TalkNet applied to the extracted output and the lip video, used to gate the output soft or hard. It isolates whether our two ingredients add anything beyond "run an active speaker detector on the output."
Ablations remove one ingredient at a time:
- gap correction ($W$ set to identity, no centering);
- self-generated hard negatives (critic trained only with interferer and time-shift negatives);
- linearity of the bridge ($W$ replaced by an MLP, testing the Ex-MCR claim in this domain);
- an MMD alignment loss in place of the linear bridge.
All systems share the same backbone checkpoints, training data, and tuning budget, with operating thresholds chosen on the same validation split.
Beating the baselines means two things at matched target-present quality, defined as SI-SDR within 0.3 dB of the ungated extractor. First, lower target-absent output power and frame-level false-alarm rate than ActiveExtract and USEV. Second, a lower area under the false-alarm/miss curve than the post-hoc TalkNet gate, both in-domain and under threshold transfer.
Why not just…?
Why not just train the extractor with zero-energy losses on target-absent mixtures? That approach is a known cost, not an untested option. In TSE-IS it lowered active-speech SDR improvement from 12.4 dB to 10.8 dB and more than doubled the failure rate (Listen only to me!). USEV's scenario losses are the audio-visual version, and we include USEV as a baseline. If our gate does not preserve target-present quality better than USEV at equal target-absent suppression, the claim fails.
Why not just run an off-the-shelf active speaker detector on the output? This is the real threat. Hallucinated speech is unsynchronized with the target's lips, so plain TalkNet may already catch most of it. Two things could make it fall short. TalkNet was never trained on separator artifacts, which can carry faint residual lip-correlated energy. And its raw scores are not calibrated across datasets. The post-hoc TalkNet baseline, together with the threshold-transfer test, settles this directly. If TalkNet matches us, the honest conclusion is that output-side synchrony checking is the contribution and our extra machinery is unnecessary. We would report it that way.
Why not just gate on the voice print, as TSE-V does? Identity checks need long enrollments to be reliable: roughly 5% equal error rate required 15–20 seconds of enrollment speech (Listen only to me!). They can also be fooled by target-like speech synthesized from noise (Towards Robust Speaker Verification with Target Speaker Enhancement). Lip synchrony needs no enrollment length at all. We test this by sweeping enrollment length from 2 to 20 seconds. We expect the lip-based gate's error to stay flat while the identity gate's error rises as enrollments shorten.
Why not just keep the modalities separate and fuse late, as Uni-Net does? Late fusion leaves its audio-cued branch exposed to exactly the hallucination we target. And Uni-Net was not evaluated on target absence (Unified Audio Visual Cues for Target Speaker Extraction). We include it in the target-absent evaluation to measure this directly, rather than assuming it.
Proposed Experiments
The main testbed is IEMOCAP-2Mix. It contains natural dyadic turn-taking with under 20% overlap and long target-absent stretches (Audio-Visual Active Speaker Extraction). VoxCeleb2-mix is used for training and for the transfer test (VoxCeleb2). We also build a new compound split from both datasets in which lip frames are occluded at 20–80% rates and the target is silent. ImagineNET never evaluated this condition, and it is where we expect audio-to-visual inpainting to reinforce hallucinations. The audio-only fallback is checked on LibriMix against the TSE-V numbers, and on SparseLibriMix for varying overlap (LibriMix).
Metrics are computed separately per scenario:
- SI-SDR and failure rate on target-present segments;
- output power in dB on target-absent and target-quiet segments;
- frame-level false-alarm and miss rates, with full curves;
- equal error rate under in-domain and transferred thresholds;
- as a downstream check, word insertions from a target-speaker recognizer on target-absent segments, the failure SQ-Whisper exhibits (SQ-Whisper).
Three results decide the proposal. First, on IEMOCAP-2Mix target-absent segments, the gated system must reach lower output power than ActiveExtract and USEV while staying within 0.3 dB of the ungated extractor on target-present SI-SDR. Failing this means output-side checking does not escape the over-suppression trade-off. Second, against the post-hoc TalkNet gate, the critic with gap correction and self-generated negatives must improve false-alarm/miss performance under threshold transfer. Otherwise the representation-gap ingredient is not pulling its weight. Third, on the occlusion-plus-silence split, the explicit fallback must hallucinate less than ImagineNET. That would confirm that keeping visual evidence independent of the audio being judged is what breaks the feedback loop.
References
- Towards Robust Speaker Verification with Target Speaker Enhancement
- Listen only to me! How well can target speech extraction handle false alarms?
- Towards understanding the modality gap in CLIP
- Neural Target Speech Extraction: An overview
- X-TaSNet: Robust and Accurate Time-Domain Speaker Extraction Network
- Speaker Extraction with Detection of Presence and Absence of Target Speakers
- USEV: Universal Speaker Extraction With Visual Cue
- Audio-Visual Active Speaker Extraction for Sparsely Overlapped Multi-Talker Speech
- ImagineNet: Target Speaker Extraction with Intermittent Visual Cue Through Embedding Inpainting
- Unified Audio Visual Cues for Target Speaker Extraction
- $C^{2}$AV-TSE: Context and Confidence-Aware Audio Visual Target Speaker Extraction
- Extending Multi-modal Contrastive Representations
- Learning Robust Visual-Semantic Embeddings
- VoxCeleb2: Deep Speaker Recognition
- LibriMix: An Open-Source Dataset for Generalizable Speech Separation
- SQ-Whisper: Speaker-Querying Based Whisper Model for Target-Speaker ASR