Audio-visual speech recognizers read the speaker's lips to recover words when the audio is drowned in noise. On the multilingual MuAViC benchmark, adding video cuts average word error rate from 70.2 to 43.1 under babble noise (MuAViC). Every multilingual system in this line was trained and tested on reasonably lit broadcast or TED video. Real deployments break that assumption. In cars, phones and street scenes the mouth region drops into darkness for a few hundred milliseconds, is washed out by glare, or flickers as the speaker passes streetlights. That happens exactly when the audio is also bad and the visual stream matters most. Nobody has measured how lip-reading gains survive these lighting failures. Nobody has measured whether robustness learned in one language carries over to others, whose mouth shapes and timing differ.
We propose a recurrent visual encoder whose memory update is controlled by an explicit, per-frame estimate of how trustworthy the lighting is. When a frame is too dark or too bright to show the lips, the encoder keeps its running summary of recent mouth movement and does not overwrite it with noise. When the frame is clean again, it resumes integrating new evidence. We test this design against strong simpler alternatives on a controlled lighting-corruption version of MuAViC. We also ask a question no prior paper has asked: does lighting robustness acquired from English training video transfer to the other eight MuAViC languages?
Core Claim
The claim has two parts, and either can fail. The first part is about mechanism. Lighting failures in real video arrive as contiguous stretches of time rather than as scattered bad pixels. For such failures, a recurrent state whose decay is set by a supervised illumination-reliability signal will lose less accuracy than two alternatives at matched size and training data. One alternative is a Conformer (a Transformer with convolution layers, the standard encoder in Auto-AVSR) trained with the same lighting augmentation. The other is the same recurrent encoder with an unsupervised gate. The advantage should grow with the length of the dropout, up to roughly word length.
The second part is about cross-lingual transfer. We hypothesize that lighting robustness learned only on English video transfers unevenly across languages. Losses should concentrate on speech sounds made inside the mouth, such as labiodental and dental fricatives. Sounds visible at the outer lip contour, such as bilabials, should suffer less. An explicit reliability signal should narrow this transfer gap more than augmentation alone, because the signal describes the camera, not the language.
Mechanism
The visual front-end stays local. It is a 3D-convolution plus ResNet-18 stem of the kind used in Auto-AVSR and Synchronous Bidirectional Learning. This respects the finding of Kreuk et al., who showed that local convolutional contrast beats recurrent context for detecting phoneme boundaries; we therefore do not ask recurrence to do the low-level work. A small head on the stem predicts a per-frame reliability $r_t \in [0,1]$.
The reliability $r_t$ then sets the decay of a diagonal linear recurrence:
$$h_t = a(r_t) \odot h_{t-1} + \big(1 - a(r_t)\big) \odot B x_t ,$$
Here $h_t$ is the visual state at frame $t$, $x_t$ is the front-end feature, $B$ is a learned input projection, and $a(r_t) = \lambda^{\,r_t}$ is an elementwise decay. The vector $\lambda$ has entries in $(0,1)$, so the state is held ($a \to 1$) when $r_t \to 0$. This is the linear recurrent unit used by BLUR, which keeps eigenvalues inside the unit circle for stability and trains with a parallel scan. We use a causal version rather than BLUR's bidirectional one, because BLUR's future context rules out streaming, which is the setting where lighting failures occur.
The reliability head is supervised during training by the known corruption parameters of synthetically degraded frames: exposure factor, clipping fraction and noise level. That supervision is what separates this design from a selective state-space layer that learns its own gates. The same $r_t$, averaged over a short window, also weights visual against audio features at fusion. This replaces the scalar heuristics used by DAVIS.
Research Program
Two pieces of prior work set the starting point.
The first motivates the problem. LRW-1000 showed that wild broadcast video, which varies in pose, resolution and lighting, cuts lip-reading accuracy to 38.19%. That benchmark never separates lighting from the other factors, so we do not treat the drop as evidence about lighting specifically. ViSpeR found different behavior on its "Wild" and "TedX" subsets but did not attribute the difference to any one factor. DAVIS built a BiLSTM lip-reader for Russian in-car commands with histogram alignment, but it reports no error figures and covers only one language.
The second constrains the cross-lingual design. Zinonos et al. found that multilingual visual pre-training helps low-resource languages, for example cutting Italian character error from 76.9% to 38.0%. They also found that it dilutes English performance, fails below roughly 30 hours per language, and transfers mostly between related languages. We therefore always report English separately and stratify results by language relatedness. XLAVS-R and MuAViC robustify only against acoustic noise. We keep their acoustic noise protocol and add a visual one.
The work proceeds in three steps.
First, we build a corruption suite. A camera-pipeline model applies exposure scaling, signal-dependent sensor noise, highlight clipping, gamma and compression to the mouth crop. It supports two kinds of corruption. Static corruptions are applied to the whole utterance. Temporal corruptions are blackout or glare segments of controlled duration, from 0 to 1 second, plus periodic streetlight flicker. These are applied to the MuAViC test sets in all nine languages and combined with MUSAN babble at the SNRs MuAViC already uses.
Second, we train the models. All models are trained on MuAViC with identical data. In the cross-lingual condition, lighting augmentation is applied only to English training clips and all other languages are seen clean.
Third, we analyze where errors arise. Error increases are attributed to viseme classes by forced alignment, which tests the predicted asymmetry between intra-oral and lip-contour sounds.
Baselines
The naive reference points are two. One is the audio-only model, which on MuAViC averages 70.2 WER at SNR 0 babble (MuAViC). The other is the unmodified audio-visual model trained on clean video and tested on corrupted video. If corrupted video pushes the audio-visual model back toward 70.2, the visual gain has been erased. That number defines how much there is to recover.
The strongest existing method is a Conformer audio-visual recognizer in the Auto-AVSR recipe. On clean English LRS3 it reports 0.9% audio-visual WER, 19.1% visual-only WER, and 5.6% WER at −7.5 dB babble (Auto-AVSR). The multilingual noisy reference is the 43.1 average audio-visual WER on MuAViC (MuAViC). No reported result was found on Lacuna for any audio-visual recognizer under controlled illumination corruption, so we establish that number ourselves. We give the Conformer the same front-end, the same corruption augmentation, a parameter count within 5% of ours, the same training steps, and the same hyperparameter search budget of equal trials per model.
The ablation that isolates the key ingredient is our recurrent encoder with the reliability supervision removed. In that variant the gate is learned freely, as in a selective state-space layer. A second ablation keeps the supervised reliability signal but feeds it to the Conformer as an extra input and as a fusion weight. This separates "explicit reliability" from "recurrent holding."
We beat the baselines if two things hold. On temporally corrupted video under SNR 0 babble, averaged over the eight non-English languages, our model must recover a substantially larger fraction of the clean-video audio-visual gain than the augmented Conformer. Across three seeds, its advantage must grow with blackout duration. Matching the Conformer on clean video is required, not optional.
Why not just…?
Why not just train a standard Conformer with lighting augmentation? This is the real threat. Attention can learn to skip bad frames, and augmentation alone may close most of the gap. Nothing in the file shows otherwise. The experiment settles it directly: the augmented Conformer is the main baseline, and the duration sweep shows whether holding state helps beyond what attention learns. If the Conformer matches us, we report that explicit recurrence is unnecessary. The cross-lingual robustness-transfer measurements remain a contribution either way.
Why not just enhance the frames first? A restoration network such as EndoUIC could be placed before a frozen recognizer. EndoUIC corrects both under- and overexposure with an illumination-prompted diffusion transformer, reaching 29.65 dB PSNR on capsule endoscopy. It is multi-step and slow, which conflicts with streaming. It also cannot restore information lost to a full blackout or to clipping. We include a lightweight single-pass enhancement front-end, plus CLAHE, as baselines. We expect them to help with mild underexposure and to fail on dropouts.
Why not just weight modalities by a reliability score at fusion, as DAVIS does? DAVIS weights audio against video using signal-condition estimates, but it reports no numbers. Fusion weighting discards the visual evidence from the frames just before and after a dropout instead of carrying it forward. The second ablation, which gives the Conformer the reliability signal at fusion, is exactly this alternative.
Why not just use an infrared or depth camera? Active sensors are lighting-invariant (depth head detection, NIR occupancy), but they are absent from consumer devices. All multilingual audio-visual corpora in the file are RGB, so cross-lingual pre-training would be unavailable.
Decisive Checks
The first check concerns synthetic realism. On real data, we compute per-frame exposure statistics for ViSpeR's "Wild" subset and for CI-AVSR in-car video, then stratify WER by those statistics. CI-AVSR reports a clean audio-visual CER of 3.48% and a noisy one of 7.99% (CI-AVSR). If our model's advantage appears only on synthetic corruptions and vanishes in the dark strata of real video, the mechanism claim fails.
The second check concerns cross-lingual transfer. In the English-only augmentation condition, we compare robustness retained in Portuguese, Spanish and Italian with robustness retained in Arabic and Russian. Robustness that does not transfer would point to language-specific learning rather than camera-level learning.
The third check is the viseme asymmetry. If intra-oral sounds do not lose disproportionately under low light, the second hypothesis is rejected as stated.
References
- MuAViC: A Multilingual Audio-Visual Corpus for Robust Speech Recognition and Robust Speech-to-Text Translation
- Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labels
- Synchronous Bidirectional Learning for Multilingual Lip Reading
- Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation
- Bidirectional linear recurrent models for sequence-level multisource fusion
- DAVIS: Driver's Audio-Visual Speech recognition
- LRW-1000: A Naturally-Distributed Large-Scale Benchmark for Lip Reading in the Wild
- ViSpeR: Multilingual Audio-Visual Speech Recognition
- Learning Cross-Lingual Visual Speech Representations
- XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception
- EndoUIC: Promptable Diffusion Transformer for Unified Illumination Correction in Capsule Endoscopy
- Fully Convolutional Network for Head Detection with Depth Images
- VPDS: An AI-Based Automated Vehicle Occupancy and Violation Detection System
- CI-AVSR: A Cantonese Audio-Visual Speech Datasetfor In-car Command Recognition