Voice activity detection decides, frame by frame, whether someone is speaking. It sits in front of speech recognition and speaker labeling, and anything it drops or wrongly lets through cannot be fixed later. Cameras help most when the microphone is swamped by noise. Lip and jaw motion still show who is talking when a siren drowns the audio. But running a face network on every video frame is expensive. It also brings its own error: models treat mouth motion as universal, so a person who chews gum, smiles broadly, breathes through an open mouth, or barely moves their lips when speaking gets misread. We propose a streaming audio-visual detector built on one asymmetry. When the audio is clean and confident, it is a free and reliable teacher. During those moments the system cheaply learns what this particular person's speaking and non-speaking mouth motion looks like. When the audio becomes ambiguous, the system runs its visual branch densely, and that branch has already been calibrated to the person.
In two sentences: listen first, and use the easy stretches of audio to label sparse video frames of the person in view, building a small personal model of "their speech motion" versus "their other motion." Spend dense visual compute only when the audio is uncertain, which is exactly when the personal visual model is needed and the audio can no longer teach it.
Core Claim
Inter-individual variability matters for visual speech detection, but not in the form studied in facial expression recognition. Work on expression personalization shows that a person's resting face (a downturned mouth, wrinkles) is misread as an active expression. It corrects this with a few unlabeled neutral frames, either by synthesizing the person's face under expressions (Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data) or by translating their features in a learned latent space (Personalized Feature Translation for Expression Recognition: An Efficient Source-Free Domain Adaptation Method). For speech detection, static resting shape is largely removed by any model that looks at motion. Optical flow, landmark trajectories, and the "dynamic images" of S-VVAD all already subtract the static face. The variability that survives is behavioral and class-conditional. Each person has their own repertoire of non-speech mouth motion and their own articulation amplitude. A neutral clip cannot reveal either; you have to observe the person both talking and not talking.
Our claim is that clean-audio moments supply exactly these labeled observations, causally and without annotation. A small per-person correction learned from them reduces visual false alarms and misses in later noisy stretches. The same audio confidence signal that selects teaching moments also decides when to run the visual encoder densely. Together these let a gated system match a dense audio-visual detector at a fraction of the visual compute, and beat it on people whose non-speech motion is unusual.
Relation to Prior Work
Three groups of work motivate the proposal. First, the front-end is the bottleneck. Downstream errors are dominated by detection mistakes in diarization and speaker-attributed recognition (A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio). Both deletions (Phone-duration-dependent long-term dynamic features for a stochastic model-based voice activity detection) and noise-triggered insertions (Speaker-dependent voice activity detection robust to background speech noise) do lasting damage. Two papers constrain our evaluation design: better standalone detectors can hurt downstream tasks (The QUT-NOISE-SRE protocol for the evaluation of noisy speaker recognition), and human-labeled boundaries hide operational error (Multimodal speaker diarization using oriented optical flow histograms). So we report insertions and deletions separately and measure downstream recognition, not only mAP.
Second, audio-visual detectors exist but are static and dense. Ava Active Speaker fuses a face tower and a spectrogram tower. LoCoNet adds long intra-speaker and short inter-speaker context. Robust Active Speaker Detection in Noisy Environments routes speech-separation features into the audio branch. Noise-Tolerant Self-Supervised Learning for Audio-Visual VAD learns noise-robust audio from lip synchrony. All of them are trained offline, then frozen, and they process every face frame. We reuse their backbones and change only how frames are scheduled and how the visual head adapts at test time.
Third, the method borrows from two places:
- Audio as teacher. Learning Visual Voice Activity Detection with an Automatically Annotated Dataset uses an audio detector to label web video offline for a population-level model. We do this online, per person, and only when the audio is confident.
- Conservative streaming adaptation. We follow the tradition of updating only from high-confidence pseudo-labels (Furui) and only for future frames (Srivastava).
The closest adaptive detectors are audio-only. HyWA generates per-speaker weight offsets once at enrollment and keeps them fixed. Robust Personal VAD refreshes a voice embedding when an auxiliary decoder sees confident single-speaker audio. Our update trigger resembles the latter, but what we update is a visual model of the person's face. The update must stay useful precisely when the audio fails, which is a different role.
Research Program
The system has three parts.
Dense audio stream. A small causal audio detector runs on every 10 ms frame and outputs a speech posterior plus a confidence estimate. It is calibrated so that confidence tracks the true error rate across signal-to-noise ratios.
Visual branch with a look-back buffer. Face crops of the tracked person are stored in a short raw ring buffer of about 0.5 s, which costs essentially nothing. The visual encoder (a TalkNet- or LoCoNet-style face tower) runs in one of two modes:
- Sparse teaching mode. When audio confidence is high, a few frames per second are encoded and labeled with the audio decision.
- Dense mode. When audio confidence drops, the encoder runs on every frame, starting from the buffered past.
The look-back is what lets sparse scheduling avoid the aliasing problem of uniform subsampling (Temporal Segment Networks). An onset buried in noise still has its preceding frames available when dense mode switches on. Query-driven keyframe selectors such as Adaptive Keyframe Sampling need future context. Our gate is causal and query-free.
Personal calibration head. The frozen visual encoder produces features $z_t$. We maintain two class prototypes for the current person: $\mu_s$ for their speaking motion and $\mu_n$ for their non-speaking motion. Each is updated only on teaching frames:
$$\mu_c \leftarrow (1-\alpha\, q_t)\,\mu_c + \alpha\, q_t\, z_t, \qquad q_t = \mathbb{1}[\text{audio confidence}_t > \tau]\cdot \mathbb{1}[\text{audio label}_t = c]\cdot \mathbb{1}[\text{sync}_t \text{ agrees}].$$
Here $c \in \{s, n\}$ is the class, $\alpha$ is a forgetting rate, $\tau$ is a confidence threshold, and $\text{sync}_t$ is an audio-visual synchrony check. The sync check ensures confident speech is attributed to the face in view rather than an off-screen talker. The personal visual score is a learned blend of the generic visual logit and the distance contrast $\lVert z_t-\mu_n\rVert^2-\lVert z_t-\mu_s\rVert^2$. The blend weight is trained offline by meta-learning over simulated per-person streams.
This head is deliberately smaller than the latent translator of SFDA-PFT. That paper's own argument against heavy pixel-level synthesis (DSFDA) carries over to streaming at even greater force. We include a translator variant as an ablation. The base encoder is never updated, which bounds drift and forgetting.
Fusion. Audio and personal-visual scores are combined with a weight driven by audio confidence. Training uses population data. Per-person streams are simulated by concatenating many clips of one identity under a noise schedule that moves from clean to noisy. This lets the meta-learned blend see realistic amounts of teaching data before it is needed.
Baselines
Naive references. The first is audio-only: the same causal audio detector with no video, which shows how much vision is needed at all. The second is dense generic audio-visual detection with no personalization, using the same encoder on every frame. Both run on identical streams.
Strongest existing method. LoCoNet reports 95.2% mAP on AVA-ActiveSpeaker, 97.2% on Talkies, and 68.4% on Ego4D. It is the strongest dense competitor because its long intra-speaker context could, in principle, absorb a person's habits within its 8-second window. We run the released configuration on the same face tracks. A causal variant restricts its context to the past so the comparison with our streaming system is fair.
Strongest noise-robust method. rASD lifts TalkNet from 84.64% to 87.97% mAP in high noise. We apply our gating and calibration on top of the same TalkNet backbone with the same noise data (RNA overlays), so any gain is additive and attributable.
Visual-only reference. S-VVAD reports 93.97% F1 on Columbia.
Ablations that isolate each ingredient. One ablation replaces audio-taught class prototypes with per-person running feature normalization, a visual analog of sequential mean subtraction (Shen et al.), using the same update budget. Another uses neutral-only calibration from the first few seconds in the style of SFDA-PFT. A third swaps confidence gating for uniform sparse sampling at matched visual compute.
What counts as winning. All variants get the same tuning budget over the shared hyperparameters. "Beating" means two things together. On per-person streams, the system must reach equal or lower total insertion-plus-deletion error than dense LoCoNet-style processing while encoding at most a quarter of the visual frames. And the full personal head must beat both normalization ablations on person-held-out, low-SNR segments by a margin larger than the run-to-run spread.
Why not just…?
Why not just run a dense audio-visual detector like LoCoNet? It may well match us on accuracy in multi-speaker movie scenes. On AVA, nobody can be calibrated because tracks are short, which is why we treat AVA as a sanity check rather than the target. The dense model has two weaknesses on long single-person streams. It pays full visual cost all the time. And its offline-trained weights cannot learn that this panelist's frequent lip-licking is not speech. The matched-compute comparison and the per-person error breakdown on RealVAD and Columbia settle both points. If dense LoCoNet beats us on idiosyncratic speakers at equal compute, the personalization claim fails.
Why not just lower the frame rate uniformly? Uniform sparse sampling misses short events that fall between samples. That is exactly the irrecoverable deletion the front-end must avoid. Our experiment reports recall on utterances shorter than 400 ms at matched frames per second. We expect uniform sampling to collapse there while audio-triggered dense mode with look-back does not. If uniform sampling at the same budget performs comparably, gating is unnecessary and we will say so.
Why not just normalize each person's visual features with a running mean, or calibrate from a neutral clip? This is the real threat, because it is nearly free. Our argument is that mean shifting corrects a static offset. A person's non-speech motions (chewing, smiling, drinking) need a class-conditional boundary that only labeled examples reveal. The expression literature itself notes that neutral-only calibration struggles with subtle states, which is the regime of low-amplitude talkers (SFDA-PFT). Both alternatives are ablations under identical budgets. If running normalization recovers most of the gain, we will adopt it as the method and reframe the contribution around gated scheduling.
Why not just personalize the audio detector, as HyWA or Robust PVAD do? Audio personalization helps reject other voices. It cannot help when the target's own voice sits below the noise floor; there, only vision carries signal. HyWA reports 85.5% mAP in unseen noise on simulated LibriSpeech+MUSAN. That setting has no video, so we cannot run it directly. Instead we compare against an audio-only dynamic-embedding baseline in our GRID streams at -5 to 5 dB.
Proposed Experiments
Datasets. Per-person streams come from three sources:
- GRID. We concatenate each of its 34 speakers' sentences and mix in CHiME4 and East Anglia noise from -5 to 20 dB, following Noise-Tolerant Self-Supervised Learning for AV-VAD. That paper reports relative AUROC gains of up to 15.33%; no absolute baseline value was found on Lacuna.
- Columbia and RealVAD panels. RealVAD's occlusions and drinking motions directly probe idiosyncratic non-speech behavior. We add RNA noise overlays, and S-VVAD's 93.97% F1 on Columbia serves as the reference.
- Ego4D. Longer egocentric conversations, against LoCoNet's 68.4% mAP.
AVA-ActiveSpeaker and Talkies serve only as a non-regression check against LoCoNet's reported numbers.
Metrics. We report frame mAP and F1, insertion and deletion rates separately, visual frames encoded per second and FLOPs, and short-utterance recall. For downstream impact, we report word error rate of a fixed recognizer on GRID transcripts after gating. No reported result was found on Lacuna for downstream recognition error under audio-visual detection on GRID, so this number is new.
Decisive checks.
- Does the mechanism explain the gain? Personalization gains should concentrate on people with frequent non-speech mouth motion or low lip amplitude, measured independently from annotated clips. If the gain is spread uniformly, it is probably generic regularization, not personalization.
- Is adaptation causal? Error on noisy segments should fall as the amount of preceding clean teaching audio grows. Shuffling teaching frames across people should erase the gain.
- Is it robust to drift? We inject an off-screen talker during clean periods and track whether the speech prototype is contaminated. We compare the synchrony check on versus off, and report false alarms in the following noisy stretch.
- Where does the compute go? We plot accuracy against the fraction of frames encoded densely, as the confidence threshold sweeps from always-dense to never-dense. This shows whether a useful operating point exists below 25% of dense compute.
References
- Disentangled Source-Free Personalization for Facial Expression Recognition with Neutral Target Data
- Personalized Feature Translation for Expression Recognition: An Efficient Source-Free Domain Adaptation Method
- S-VVAD: Visual Voice Activity Detection by Motion Segmentation
- A Comparative Study of Modular and Joint Approaches for Speaker-Attributed ASR on Monaural Long-Form Audio
- Phone-duration-dependent long-term dynamic features for a stochastic model-based voice activity detection
- Speaker-dependent voice activity detection robust to background speech noise
- The QUT-NOISE-SRE protocol for the evaluation of noisy speaker recognition
- Multimodal speaker diarization using oriented optical flow histograms
- Ava Active Speaker: An Audio-Visual Dataset for Active Speaker Detection
- LoCoNet: Long-Short Context Network for Active Speaker Detection
- Robust Active Speaker Detection in Noisy Environments
- Noise-Tolerant Self-Supervised Learning for Audio-Visual Voice Activity Detection
- Learning Visual Voice Activity Detection with an Automatically Annotated Dataset
- Automatic Speech Recognition and its Application to Information Extraction
- Online speaker adaptation and tracking for real-time speech recognition
- HyWA: Hypernetwork Weight Adapting Personalized Voice Activity Detection
- Robust Personal Voice Activity Detection for Mitigating Domain Mismatch and False Acceptance Scenarios
- Temporal Segment Networks: Towards Good Practices for Deep Action Recognition
- Adaptive Keyframe Sampling for Long Video Understanding
- Speaker-Independent Continuous Mandarin Speech Recognition Under Telephone Environments