Let the Lexicon Pick the Language: Shared Phone Bottlenecks for Code-Switched Lipreading

Lipreading, meaning recognizing speech from silent video of the mouth, is hard because many sounds look identical on the lips. When a speaker mixes Mandarin and English in one sentence, the problem gets worse. An English word can look exactly like a Mandarin one: a recent benchmark reports English "new" read as Mandarin "午" and "match" read as "没吃". Current code-switched lipreaders try to make the video encoder decide, frame by frame, which language is being spoken. We argue this asks the encoder for something the pixels often do not contain. Instead, the model should output only what the lips can show: a language-neutral sequence of sound classes. A bilingual pronunciation dictionary and a language model then decide both the words and the language they belong to.

The same design addresses a second problem. Lipreading systems need to work with large, open vocabularies, not just a fixed set of words seen in training. A model that predicts shared sound units can recognize a word it never saw on video, as long as the word is in the dictionary. We test both claims on top of a self-supervised audio-visual encoder, one pretrained on unlabeled talking-face video to predict hidden-frame cluster identities. This combination has not been tried on code-switched video.

Core Claim

On code-switched lipreading, a self-supervised visual encoder trained to emit a single shared, tone-free phone inventory should make fewer cross-lingual substitution errors than the same encoder trained to emit characters or subwords with language routing. Cross-lingual substitution means outputting a word in the wrong language. The phone inventory covers Mandarin initials and finals plus English phonemes, merged wherever their visible articulation coincides. Language identity is assigned only during decoding, by a weighted finite-state transducer (WFST) that composes a bilingual lexicon with a code-switched language model. A WFST is a graph that combines these knowledge sources into one search space. The second prediction is that the shared-phone model recovers words absent from the video training set far better than any character- or subword-output model. The first prediction concerns language confusion, the second vocabulary growth. Both follow from removing the language decision from the visual model.

Grounding in Prior Work

Three findings motivate the design. Large-Scale Visual Speech Recognition introduced Vision-to-Phoneme (V2P), which maps video to phoneme distributions and decodes them with a finite-state lexicon and 5-gram model. Its argument was that this keeps visual ambiguity separate from spelling ambiguity. The system reached 40.9% WER on LSVSR, but it is English-only and supervised. Imagined-speech EEG work by Saha et al. found the same benefit from composing words out of phonological sub-units. Language Without Borders built the Chinese-English code-switching lipreading benchmark (CSLR) and documented the visual cross-lingual confusions. Its best model is a language-adaptive mixture-of-experts (LR-MOE) with an auxiliary language-identification loss, trained on small supervised backbones. That model forces language discrimination into the encoder, which is exactly the design we question.

Two lines of work supply the method. Multilingual Visual Speech Recognition with a Single Model by Learning with Discrete Visual Speech Units and Efficient Training for Multilingual Visual Speech Recognition provide the multilingual self-supervised encoder (mAV-HuBERT) and a cheap discrete-unit training path. Both evaluate only single-language utterances. Boosting End-to-End Multilingual Phoneme Recognition through Exploiting Universal Speech Attributes Constraints shows that a shared cross-language attribute inventory can constrain phone recognition. We adapt that idea to what is visible on the lips.

Audio code-switching work pulls in the opposite direction, and we address it directly. CAMEL, Adapting Whisper for Code-Switching through Encoding Refining and Language-Aware Decoding, the Language Alignment Loss and gated language experts all gain by building language awareness into the encoder or decoder. That works in audio because acoustics carry real language cues. Tones, vowel quality and rhythm are audible but largely invisible on the lips. Our hypothesis is therefore modality-specific: the routing that helps audio should help video much less. The experiments measure this rather than assume it.

A further constraint comes from audio self-supervised models. Fan et al. find that language-discriminative information sits in lower layers, with phonetic content higher up. We therefore probe mAV-HuBERT layers for language identity on CSLR before choosing where to attach the phone head. The probing result is itself a finding: if visual layers carry little language signal, that supports our premise.

Research Program

We fine-tune mAV-HuBERT on CSLR with a connectionist temporal classification (CTC) head over the shared inventory. CTC is a training loss that aligns frame outputs to label sequences without frame-level timing. Training targets come from transcripts passed through a pinyin lexicon and an English pronunciation dictionary, so no frame-level labels are required.

The inventory is built in three steps. We start from IPA-mapped units for both languages and strip Mandarin tones, since tone is not visible on the lips. We then merge cross-language units whose visible articulation matches, using two sources of evidence: confusion statistics of a held-out phone recognizer and phonetic place and rounding features. Decoding composes the CTC topology, the bilingual lexicon (every word tagged with its language), and an n-gram language model. The n-gram model is trained on CSLR training text plus monolingual Mandarin and English text. An optional neural rescoring pass follows.

One known risk shapes the analysis: mAV-HuBERT's pretraining languages may not include much Mandarin. We therefore also report a variant that continues self-supervised pretraining on CSLR video before fine-tuning.

Evaluation uses CSLR's 21,224 test clips, scored with character error rate (CER), word error rate (WER) and mixed error rate (MER). MER scores Mandarin by character and English by word. We add three diagnostics. The first is the cross-lingual substitution rate, the fraction of errors that output the wrong language. The second is the error rate in a ±0.3 s window around language switches. The third is recall on a held-out-vocabulary split: we remove from training every clip containing a chosen set of 300 English and 300 Mandarin words, then test on clips containing them. As a sanity check on English-only transfer, we also report zero-shot LRS3-TED WER, where V2P reported 55.1% in Large-Scale Visual Speech Recognition.

Baselines

The naive reference point is mAV-HuBERT fine-tuned with a plain CTC head over Mandarin characters plus English subwords, using no language machinery and greedy decoding. It tells us how much the pretrained encoder alone buys on code-switched video.

The strongest existing method is LR-MOE from Language Without Borders. It reports CER 33.01%, WER 45.37% and MER 37.54% on CSLR, and it is the only published system built for this exact task. We compare against the reported number. We also re-implement LR-MOE's routing and language-identification loss on top of the same mAV-HuBERT encoder, so that encoder quality is separated from output design. That re-implemented version gets the same training data, the same number of GPU-hours and the same hyperparameter search budget (twenty trials) as our model. It also receives the same n-gram language model through shallow fusion, so the comparison is not unfair through decoding. Beating it means a lower MER with non-overlapping speaker-bootstrap confidence intervals, together with a lower cross-lingual substitution rate. A gain in MER alone, without fewer wrong-language errors, would not support our mechanism.

The key ablation keeps everything fixed but uses language-tagged phones: English /n/ and Mandarin /n/ become separate units, which forces the encoder to choose a language. If shared phones do not beat tagged phones on cross-lingual substitutions, our central claim fails. A second ablation restores Mandarin tones, testing whether asking the lips for invisible information hurts. A third trains the shared-phone head on 1,000-unit discrete mAV-HuBERT tokens instead of continuous features. This tests whether the cheap discrete path loses the cues that mark a switch, an open question raised by the discrete-unit papers' own need for continuous fine-tuning.

Why not just…?

Why not just fine-tune mAV-HuBERT end to end on characters with a language-ID loss? This is the real threat. A large pretrained encoder plus enough CSLR data might learn language cues we believe are absent, such as jaw rhythm or the speaker's habits. The audio results in CAMEL and the Language Alignment Loss show that this approach works when the signal carries language information. The re-implemented LR-MOE baseline on the same encoder is exactly this alternative. If it matches us on in-vocabulary MER, our claim narrows to vocabulary growth and data efficiency, and we would report it that way. The held-out-vocabulary split is where a character model has no route to unseen Mandarin characters in new words or unseen English spellings, and where the difference should be largest.

Why not just let a large language model correct the output of a code-switched lipreader? The authors of the Language Alignment Loss found that LLM error correction tends to hallucinate or choose the wrong language path unless given grounded language hints. In lipreading, those hints are exactly what is unreliable. We run LLM N-best correction on top of both the baseline and our system. If correction closes the gap entirely, the lexicon-based decoder adds little. We expect it does not, because a character lattice that has already committed to the wrong language gives the LLM fewer correct candidates to choose from.

Why not just use discrete visual units, which are ten times cheaper to train on? The discrete-unit work reports the speedup but evaluates only single-language speech. It also relies on continuous fine-tuning to recover fine motion detail. The discrete-token ablation measures whether quantization collapses the cross-language pairs that matter here. If it does not, we adopt discrete units as the cheaper path, which would itself be a useful result.

Decisive Checks

Three results would settle the proposal. The first is the cross-lingual substitution rate of shared phones versus language-tagged phones on the same encoder. A clear reduction supports the claim that the encoder should not be asked to pick a language. No reduction falsifies it. The second is held-out-vocabulary word recall. We predict the shared-phone system recovers a substantial fraction of unseen words while character and subword systems recover few. The third is a learning curve at 10%, 25% and 100% of CSLR training data. If the language decision really belongs in the decoder, our advantage should be largest at small data, since the decoder's knowledge comes from text rather than video.

Layer probing for language identity on mAV-HuBERT gives the mechanistic check. If language identity is nearly undecodable from visual features at switch points, the case for moving the decision into the lexicon is made directly. If it is easily decodable, we expect the routing baseline to win, and we will say so.