Cued Speech is a visual system for deaf communication. The speaker's hand shape encodes consonants and hand position encodes vowels, and both supplement what the lips show. The hand reliably moves ahead of the mouth: about 140 ms early for vowels and about 60 ms for consonants (Liu et al.). This lead varies with speaking rate, with position in the sentence, and with whether the cuer is hearing-impaired (Gao et al.). Recognition systems currently handle the lead in one of two ways. One is to shift the hand stream by an amount predicted from lip timing. This is interpretable, but it runs as a separate hand-engineered stage before decoding. The other is to let a large multi-stream network absorb the misalignment implicitly (the 3D-reconstruction multi-stream model). This gives low error, but it says nothing about the timing and needs scarce labelled video.
The idea here borrows one working habit from multi-orbital condensed-matter physics. In that field, a complicated system is described by a small matrix of coupling parameters between a few named channels (the analogue of "inter-orbital hopping"). Researchers then test explicitly whether the matrix splits into independent blocks, and whether parameters fitted on small, clean systems carry over to large, messy ones. We propose to model the hand lead as the output of such a coupling matrix over the timing of the hand-shape, hand-position and lip streams. We would build it as a small differentiable alignment layer inside an end-to-end recognizer. We then test two things: whether the matrix has the block structure the phonology predicts, and whether coupling parameters fitted on isolated words and a few cuers transfer to continuous sentences and unseen cuers.
Core Claim
The claim is that hand–lip lead time is a low-parameter coupled system. The lead for a given cue depends on the class of the cue and on the timing of the neighbouring streams. It does not depend on the class alone. The claim also says the coupling is transferable, so a model that encodes this structure should match implicit alignment when data are plentiful and beat it when they are not. This is most visible for unseen cuers, hearing-impaired cuers, and training on isolated words only. The physics does not supply dynamics here. We are not claiming that articulators obey a Hamiltonian. It supplies a method:
- Name the channels.
- Fit a sparse coupling matrix.
- Measure how much of each observable comes from off-block terms. This is the analogue of the one-third $d_{z^2}$ weight at the Fermi level in cuprates (Matt et al.).
- Treat the failure of a truncated description as a measurable quantity rather than an assumption.
Research Program
The lead model is written for cue $i$ with phonological class $c(i)$ (vowel/position or consonant/shape):
$$\hat\tau_i = \tau_{c(i)} + \sum_{\beta} J_{c(i)\beta}\, x_{i\beta},$$
The terms are defined as follows. $\tau_{c}$ is a class-level baseline lead. Each $x_{i\beta}$ is a standardized timing feature of stream $\beta$ near cue $i$. These features are the lip-vowel interval, duration and distance-to-end features of Gao et al., plus hand-shape and hand-position transition times estimated from pose. $J$ is the coupling matrix.
The predicted lead is applied as a differentiable soft shift: the hand features at frame $t$ are resampled at $t-\hat\tau_i$ with a narrow Gaussian kernel. The shifted streams then feed the same encoder and CTC decoder (connectionist temporal classification, a loss that aligns frame outputs to label sequences without frame-level labels) used by the multi-stream model. The whole system is trained end to end, and $J$ gets a sparsity penalty.
This design differs from both existing lines of work. Gao et al. fit a piecewise linear regression of lead on lip metrics as a separate preprocessing stage. It uses segmented vowels and has no cross-stream coupling from hand-shape timing. Our layer is trained through the recognition loss and couples all three streams. Liu et al. use two fixed class-level delays, which corresponds to the special case $J=0$.
The analysis rests on three physics-style questions, and each is falsifiable.
Decoupling. Is $J$ block-diagonal, with the vowel/position sector independent of the consonant/shape sector, in the way Jiang et al. found hidden decoupled orbital groups in FeGe? We compare a full $J$ against a block-restricted $J$. We report the fraction of lead variance explained by off-block terms.
Transfer. Do parameters fitted on isolated words carry over to continuous speech? This follows the transferable parameters of the twisted-bilayer workflow, which fits on small cells and extrapolates to the magic angle. We fit $J$ on the French isolated-word corpus and freeze it for continuous sentences. On Mandarin, we fit on normal-hearing cuers and apply to hearing-impaired cuers, with only $\tau_c$ refit.
Truncation. When a third stream is dropped, does the two-stream lead estimate become biased in a predictable direction? This is the analogue of two-band models failing when a third pocket crosses the Fermi level (Zeng et al.).
As a small, secondary scientific check, we apply the same regression form to the Brazilian Portuguese rt-MRI measurements of Meireles et al.. There, velum lowering in nasal syllables is locked to consonant release (34–44 ms after it) rather than to vowel onset (107–146 ms after it). Under our model, the difference between "event-triggered" and "in-phase" timing is simply which entry of $J$ is nonzero. So the model should recover the release coupling for nasalized stops and the vowel-onset coupling for oral syllables from one shared fit. This dataset has two speakers and twelve words. It is a sanity check of the representation, not a benchmark.
Baselines
Naive reference points. The do-nothing reference is the multi-stream recognizer with unshifted hand streams. The next is a single global shift of the hand stream, tuned on validation. We also run the two fixed class delays of Liu et al., who report continuous phoneme correctness of 76.63% against 72.04% for their prior architecture in an HMM-GMM (hidden Markov model with Gaussian mixture) pipeline. We re-implement these delays inside our neural decoder so that the backbone is the same.
Strongest implicit competitor. This is the multi-stream model of the 3D-reconstruction multi-stream paper. It combines a modulated graph convolutional network over MediaPipe and PIXIE joints with an RGB stream and CTC plus KL-divergence auxiliary losses. It reports a phoneme error rate of 8.35% on the French Cued Speech dataset. It is the strongest because it already reaches low error with no explicit alignment. Our layer is inserted into exactly this backbone, trained on the same 979/108/108 split, with the same number of epochs and the same hyperparameter-search budget. Only the alignment layer differs. For British English, the file's excerpt is truncated, so no reported result found on Lacuna for the British English Cued Speech benchmark. We rerun the backbone ourselves under 5-fold cross-validation.
Strongest explicit competitor. This is the lip-metric regression of Gao et al.. It reports 96.06% vowel recognition accuracy on the roughly 4,000-video Mandarin multi-cuer corpus, against 96.56% with ground-truth alignment. Because that leaves only half a point of headroom in the pooled setting, we do not expect to beat it there. The comparison that matters is leave-one-cuer-out and normal-to-hearing-impaired transfer, using their regression retrained under the same splits.
Ablations. We keep the soft-shift layer but vary the lead model. In one version the lead is a free learnable per-class scalar. In another, a small unconstrained multilayer perceptron predicts the lead from all features. A third uses block-diagonal $J$ and a fourth uses full sparse $J$. These isolate whether cross-stream coupling, rather than differentiable shifting as such, carries any gain.
Beating the strongest competitor means one of two outcomes. The first is equal pooled error, within seed variance over five seeds, together with a statistically reliable reduction in phoneme error on held-out cuers, hearing-impaired cuers or the word-to-sentence transfer. The second is a pooled gain when training data are subsampled to 25% and 10%.
Why not just…?
Why not just let the end-to-end network learn asynchrony implicitly? This is the real threat. With the full French training set it may already be near ceiling. The multi-stream paper itself notes that annotated Cued Speech data are scarce, which is exactly the regime where explicit structure should help. The subsampling curves and the held-out-cuer splits settle this. If the implicit model matches ours at 10% data and on unseen hearing-impaired cuers, the proposal's practical claim fails. Only the interpretability of $J$ would remain.
Why not just run Gao et al.'s regression as a preprocessing step? It already works well on pooled Mandarin data. However, it needs vowel segmentation, is fitted outside the recognition loss, and was reported to vary with hearing status (Gao et al.). The normal-to-hearing-impaired transfer experiment compares it head to head with our model under identical splits. If it transfers as well as ours, the novelty reduces to end-to-end training and we would say so.
Why not just learn a free per-class shift or an unconstrained lag predictor? If a per-class scalar matches full $J$, then the lead really is class-determined. That would mean the multi-orbital lens adds nothing beyond Liu et al.. An unconstrained predictor could fit the pooled data as well as $J$ does. The test is whether it transfers from isolated words to sentences, where the distance-to-end effect only appears in continuous speech. The ablation reports both outcomes directly.
Proposed Experiments
French Cued Speech. We report phoneme error for the full backbone against the backbone plus our layer, over five seeds and at 100%, 25% and 10% of training data. A separate experiment fits $J$ on the isolated-word corpus (50 words, 10 repetitions), freezes it, and evaluates continuous sentences from the LM corpus.
Mandarin multi-cuer corpus. We report vowel and phoneme accuracy under leave-one-cuer-out across the five cuers. A separate train-on-three-normal-hearing, test-on-two-hearing-impaired split compares our layer against Gao et al.'s regression and the implicit backbone.
Interpretability. For every fitted $J$, we report the off-block variance fraction with bootstrap confidence intervals. We also report the mean absolute error of predicted lead against annotated lead where segmentations exist, and the direction of bias when one stream is removed.
Decisive checks. The proposal is judged on three results. First, our layer must beat the per-class-shift ablation on at least one transfer split. Second, it must not lose to the implicit backbone on pooled French data. Third, the rt-MRI check must recover the release-locked velum coupling in nasal syllables. If the first fails, the coupling idea is wrong for Cued Speech. If only the second fails, the structure costs accuracy and has value only as an analysis tool.
References
- Re-Synchronization Using the Hand Preceding Model for Multi-Modal Fusion in Automatic Continuous Cued Speech Recognition
- A Novel Interpretable and Generalizable Re-synchronization Model for Cued Speech based on a Multi-Cuer Corpus
- A Multi-Stream Framework Utilizing 3D Human Reconstruction for Cued Speech Recognition
- Direct observation of orbital hybridisation in a cuprate superconductor
- FeGe as a building block for the kagome 1:1, 1:6:6, and 1:3:5 families: Hidden <mml:math xmlns:mml="http://www.w3.org/1998/Math/MathML"><mml:mi>d</mml:mi></mml:math>-orbital decoupling of flat band sectors, effective models, and interaction Hamiltonians
- Construction of low-energy symmetric Hamiltonians and Hubbard parameters for twisted multilayer systems using ab initio input
- Compensated Semimetal LaSb with Unsaturated Magnetoresistance
- Gestural coordination of Brazilian Portugese nasal vowels in CV syllables: A real-time MRI study