Agreement-Gated Early Fusion for Unseen Noise

Audio-visual speech recognizers read lips alongside listening, and they are most useful when the audio is noisy. The best results come from early fusion, where audio and video features are mixed inside the encoder rather than only at the output. Early fusion has a known weakness, though. When the audio is badly corrupted, mixing lets noisy audio features contaminate the clean visual ones. Systems that fix this learn a "reliability gate" that turns down the audio when it looks bad. A gate learned from training noises, however, may simply memorize what those noises look like. This proposal tests one concrete claim. If the gate uses a signal that carries no information about the noise itself, early fusion should transfer to noise types never seen in training. The signal we propose is how well the audio agrees with what the lips predict it should sound like. A gate learned from the audio features should not transfer in the same way.

In two sentences: we move reliability gating into the early-fusion encoder, compute it only from audio-visual disagreement, and restrict it to two scalar parameters so it cannot learn noise-specific cues. We then test, under a strict held-out-noise-family protocol, whether this gate stays calibrated on unseen babble, music, machine noise, reverberation and telephone-band audio, where learned gates and plain noise augmentation do not.

Core Claim

Three pieces of prior work set up the question. Attentive Fusion Enhanced Audio-Visual Encoding motivates it. That paper compares fusion placements in a Transformer recognizer and finds that encoder-level fusion beats late fusion: 48.16% versus 57.79% word error rate (WER) at −5 dB babble. It also finds that symmetric two-way cross-attention collapses as the signal-to-noise ratio (SNR) drops, because noisy audio pollutes the video stream. Its one unseen noise type (buccaneer2) is too thin to say anything about generalization.

Router-Gated Cross-Modal Feature Fusion supplies our method. It builds a label-free, token-level reliability score by comparing real audio embeddings against audio embeddings predicted from video with a cross-modal masked autoencoder. That paper places the gate only in the decoder, lets it feed a learned global gate, and assumes clean video. Robust Self-Supervised AVSR, the noise-augmented AV-HuBERT model, constrains our design. It shows that static early concatenation plus heavy MUSAN augmentation is already very strong. It also reports a mismatch penalty between synthetic and natural noise, so any gain we claim must survive comparison with better augmentation.

Our claim differs from all three in one respect. The source of the reliability signal, and the capacity of the gate reading it, determine whether early fusion generalizes across noise types. The research file names the relevant failure "noise feature entanglement": trained on a narrow set of noises, networks use the noise statistics as predictive features. A gate that reads the audio features directly is exposed to exactly this failure. A gate that reads only an agreement score between audio and lip-predicted audio, through a monotone two-parameter map, is not.

Research Program

We start from a pre-trained AV-HuBERT encoder and add an asymmetric fusion block after its first few layers. The placement follows the early-fusion design of Attentive Fusion. At each frame $t$, a frozen cross-modal predictor produces $\hat a_t$, the audio embedding predicted from a short window of lip features. We define reliability as

$$r_t = \sigma\big(\alpha\,(\cos(a_t,\hat a_t) - \tau)\big),$$

where $a_t$ is the observed audio embedding, $\sigma$ is the logistic function, and $\alpha,\tau$ are the only two learned parameters. Visual tokens then receive audio information scaled by $r_t$:

$$h^v_t \leftarrow h^v_t + r_t\,\mathrm{Attn}(h^v_t, H^a).$$

Audio tokens always attend to video. $H^a$ denotes the audio token sequence, and $\mathrm{Attn}$ is the cross-attention used in the Attentive Fusion AV-cross variant. The design keeps the useful direction, lips cleaning up audio, and closes the polluting direction only when agreement is low. In clean conditions it recovers AV-cross, which the Attentive Fusion authors found best at high SNR. In noise it degrades toward the one-way AV-align variant.

The agreement score has an obvious flaw. Low agreement can mean the video is bad (occlusion, a profile view, blur) as easily as the audio. The Router-Gated paper sidesteps this by assuming clean video. We handle it by also computing the reverse residual, video predicted from audio. The gate closes the audio→video path only when the audio-side residual is anomalous relative to a running per-utterance baseline. We measure this failure directly rather than assume it away.

Baselines

The naive reference point is an audio-only AV-HuBERT-style model fine-tuned with the same noise augmentation. It tells us how much any use of video buys under unseen noise. Robust Self-Supervised AVSR reports that audio-visual models give 53.0% and 70.8% relative WER reductions over audio-only ASR in noise, with 30 and 433 labeled hours respectively. We expect to reproduce a gap of that order on seen noise, and we measure how much of it remains on held-out families.

The first strong competitor is static early concatenation: noise-augmented AV-HuBERT itself, fine-tuned on the same LRS3 split. It is the architecture most practitioners would deploy.

The strongest existing method is the decoder-placed router gate of Router-Gated Cross-Modal Feature Fusion. That paper reports LRS3 WER at 0 dB babble falling from 35.92% (AV-HuBERT) to 21.40%. We will run it with its published design on the same backbone, the same labeled hours, the same noise pool and the same number of fine-tuning steps. The hyperparameter search budget will be matched across all methods.

Beating this baseline has a specific meaning here. A tie on seen noise is acceptable. On held-out noise families, we need a lower WER averaged over SNRs of −5 to 20 dB. We also need a smaller generalization gap, meaning WER on the held-out family minus WER on seen families at matched SNR, with the difference exceeding seed-to-seed variation across three runs.

Two ablations isolate the key ingredient. The first keeps our encoder placement and asymmetric block but replaces the two-parameter agreement gate with a learned gate: a small network reading the audio tokens and trained end-to-end. If this matches us on unseen noise, the claim about the reliability signal is false. The second uses the agreement signal but lets a learned multilayer network map it, together with the audio features, to the gate. This tests whether low gate capacity is what matters.

Why not just…?

Why not just augment with more noise types? This is the most serious threat. Robust Self-Supervised AVSR shows how far augmentation goes. If coverage alone closes the gap, our gate is unnecessary. We settle it by varying how many noise families are available in training, from one up to all but the held-out family. We then plot the generalization gap against the number of families for both static AV-HuBERT and our model. If the two curves converge quickly, augmentation wins and we will say so. The interesting outcome is a persistent gap for families that differ structurally from everything seen, namely reverberation and band-limiting.

Why not just use one-way AV-align fusion, which never lets audio touch video? Attentive Fusion found that AV-align is safe in noise but loses to AV-cross in clean and high-SNR audio. Our block should match AV-cross at high SNR and AV-align at low SNR. AV-align is therefore run as a baseline. The win condition is being no worse than the better of the two at every SNR.

Why not just keep the router in the decoder, where it already works? A decoder gate leaves the encoder's visual representations exposed to whatever the audio did in earlier layers. We test whether this matters with a layer-wise linear probe that reads phonetic information from the visual tokens only, evaluated on unseen noise. If visual-token probe accuracy under decoder gating matches ours, encoder placement adds nothing.

Why not just gate on an estimated SNR? An SNR estimator is itself trained on noise examples, so it inherits the same entanglement risk. We include it as a gate input in the learned-gate ablation.

Proposed Experiments

The training data are LRS3 speech with its 30-hour and 433-hour labeled splits, mixed with MUSAN (speech, music and natural-noise families) and NOISEX-92 (babble, factory, buccaneer, tank, machine gun). We use a leave-one-family-out protocol, so every model is tested on a family whose samples and category were both absent from training. Two structured shifts are applied only at test time: simulated room reverberation and 8 kHz telephone-band filtering, the channel mismatch highlighted in the file's discussion of telephony systems. LRS2 serves as an out-of-domain speech-and-video test. The Router-Gated paper reports only a qualitative lead there; no reported LRS2 WER was found on Lacuna, so we will produce our own. Visual robustness is probed with synthetic occlusion and blur on the lip region.

Because every mixture is synthetic, frame-level local SNR is known exactly. This supports the decisive check: the rank correlation between the gate value $r_t$ and local SNR, measured on held-out families. The hypothesis predicts that this correlation stays near its seen-noise value for the agreement gate but drops for the learned gate. If both hold up, WER differences cannot be attributed to the signal source. If neither does, the gate is not doing what we claim.

The second decisive check is the visual-corruption test. When lips are occluded and audio is clean, the gate must not close the audio path. If it does, the agreement signal is ambiguous in a way the reverse residual fails to fix, and the method is limited to the clean-video regime already assumed by the decoder router.