Electromechanical Delay as Free Preview for Real-Time Silent Speech Voicing

Muscles fire before the mouth moves. In facial surface electromyography (sEMG), which records the electrical activity of the speech muscles through skin electrodes, the electrical signal leads the actual articulator movement by roughly 50 ms. This lag is called the electromechanical delay. Every real-time "silent speech interface", a system that turns mouthed but unvoiced speech into synthetic audio, has treated this lag as a nuisance to be aligned away with a fixed time shift. We propose to treat it as a resource. If the signal that drives speech arrives ahead of the speech, a decoder that runs strictly on the past can still predict the sound the mouth is about to make. It can then play that sound at, or slightly before, the moment of articulation, even after paying for computation, vocoding, and wireless transmission.

The core proposal is a streaming EMG-to-speech decoder that emits, at every step, a short packet of predicted acoustic frames for several future horizons. A playback stage then selects the frame whose horizon matches the delay actually experienced. This borrows the "action packet indexed by arrival time" idea from delayed reinforcement learning and applies it to audio. It matters because the users who need these systems are laryngectomy patients, people in noisy or confidential workplaces, and people who will not speak aloud in public. They need to hear their own synthesized voice without perceptible lag. Without that auditory feedback, silent articulation drifts away from normal speech, and decoders trained on vocalized speech fail.

Core Claim

Our claim has two parts, both testable. First, for sEMG speech decoding there is a measurable "free preview" window. Within it, predicting the acoustic frame $h$ milliseconds ahead of the latest EMG sample costs little accuracy relative to predicting the present frame, and accuracy should stay roughly flat until $h$ approaches the electromechanical delay. Second, a decoder that predicts several horizons at once and picks the right one at playback time keeps intelligibility under computation and transmission jitter, while matching or beating a causal decoder with a fixed shift. When perceived latency is held equal, it should also beat chunked decoders that wait for a small amount of future signal.

The payoff is a closed loop. We test whether latency-free self-generated auditory feedback moves silent articulation back toward audible articulation. This is an effect previously shown only with open-loop playback of pre-recorded sentences.

Where This Sits

Three prior systems motivate the gap and define what is not new. Diener and Schultz in Investigating Objective Intelligibility in Real-Time EMG-to-Speech Conversion built a real-time EMG-to-mel-spectrogram network. They found that objective intelligibility peaks with a fixed ~50 ms EMG-to-audio offset, which they attribute to electromechanical delay, and degrades beyond 75–100 ms, at about 79 ms total system latency. They spend the electromechanical delay on alignment. We ask how much of it can be spent on prediction instead, and whether the offset should be estimated online rather than fixed.

CSL-EMG_Array introduced strictly causal features and running normalization, and documented quality decay across sequential evaluation blocks. That work supplies our causal front-end and our drift testbed, but it never predicts forward. The EMA pilot study Real-time control of a DNN-based articulatory synthesizer for silent speech conversion targeted sub-50 ms feedback from glued, invasive articulograph coils using a reactive window of 10 past frames. We work with a wearable modality and predict ahead rather than react.

The most accurate voicing systems constrain the design in a different way. Digital Voicing of Silent Speech, An Improved Model for Voicing Silent Speech, and Confidence-Based Self-Training for EMG-to-Speech use bidirectional models over whole utterances. They reach open-vocabulary word error rates of 68.0%, 42.2%, and 21.87% respectively on the Gaddy & Klein data. None of them can stream. We reuse their training-time trick, which aligns silent EMG to parallel vocalized audio with dynamic time warping to obtain targets. Alignment is allowed to be offline because only inference must be causal. Their numbers serve as the acausal ceiling.

The control literature supplies the method. Predictor feedback in Event-Triggered Stabilization of Nonlinear Systems with Time-Varying Sensing and Actuation Delay projects state forward to the moment an action takes effect. The asynchronous controller in Data Efficient Reinforcement Learning for Legged Robots. plans from the state predicted at compute completion. Adaptive Reinforcement Learning for Unobservable Random Delays sends a matrix of actions, one row per possible arrival time, and reports ~8368 versus ~4159 return for a buffering-style baseline on HalfCheetah under real WiFi delays. Our audio packet is the direct analogue.

Regret of $H_\infty$ Preview Controllers shows that a few steps of preview nearly close the gap to a fully non-causal controller in a linear system. This suggests that a short look-ahead may recover much of what the bidirectional models gain. Electromechanical delay may provide that look-ahead without waiting.

The motor-control evidence explains why the loop matters. Forward dynamic models in human motor control shows that people estimate limb state by combining a forward model with delayed sensory feedback. Impact of different feedback mechanisms in EMG-based speech recognition found that silent speech without feedback yields ~90–95% word error rate. Synchronized playback of pre-recorded audio cut that error by 2.2–17.2% relative, lengthened utterances, and shifted EMG spectra toward audible speech. That feedback was open-loop. Ours is generated from the user's own ongoing muscle activity, which no prior EMG work has done at near-zero latency.

Research Program

The decoder is a causal convolutional front-end followed by a causal Transformer, reusing the residual blocks and auxiliary phoneme loss of Gaddy & Klein 2021 with attention masked to the past. A multi-horizon head predicts mel frames $\hat{y}_{t+h}$ for horizons $h \in \{-40, -20, 0, 20, 40, 60, 80\}$ ms relative to the newest EMG sample at time $t$. The head also predicts a per-horizon variance. Training minimizes the summed per-horizon spectral loss plus the phoneme loss. Targets come from DTW-aligned parallel vocalized audio for silent utterances.

At run time, a playback scheduler measures the elapsed computation and transmission delay $d$ when a packet arrives. It plays the frame with $h$ closest to $d + \delta$, where $\delta$ is a user-specific articulation offset. A small online estimator re-fits $\delta$ during the session by maximizing cross-correlation between predicted onsets and EMG burst onsets. This is the "reset by delayed measurements" step of predictor feedback. It is meant to track fatigue and electrode drift without retraining the network. A neural vocoder runs causally on the selected frames.

The first deliverable is a measurement: the preview curve, giving accuracy as a function of $h$, per speaker, and broken down by phoneme class. This curve shows whether the free-preview window exists and how wide it is. The second deliverable is the jitter-robust streaming system. The third is a closed-loop user study.

Baselines

The naive reference point is a causal decoder with the same architecture and no shift, which predicts the acoustic frame aligned with the newest EMG sample and plays it when ready. All perceived latency then equals compute plus transmission. Every improvement must be measured against this.

The strongest real-time competitor is a fixed-offset causal decoder in the style of Diener and Schultz. We retrain it with our backbone, so it differs only in having a single horizon set by validation sweep, and it learns its shift end to end. The file gives its operating point as a ~50 ms optimal offset and ~79 ms total latency. No reported STOI or WER value for it on the Gaddy & Klein or CSL-EMG_Array benchmarks was found on Lacuna. We will therefore report it ourselves, and it is the comparison that decides the jitter claim.

A chunked streaming decoder that waits for $k$ future frames, with $k$ swept, represents the "buy accuracy with latency" option. It is compared at matched perceived latency.

The acausal ceiling is the reported 21.87% open-vocabulary WER of CoM2S and 42.2% of Gaddy & Klein 2021 on the Gaddy & Klein dataset. We also retrain the 2021 model ourselves, so that its WER is measured with the same ASR scorer as our systems.

Two ablations isolate the key ingredients. The first keeps the multi-horizon head but uses fixed $\delta$. The second keeps online $\delta$ but uses a single horizon.

All systems share the training data, the DTW-derived targets, the same parameter count within 10%, and the same hyperparameter search budget. They are evaluated with the same ASR model on the synthesized audio. Beating the fixed-offset baseline means lower WER at equal mean perceived latency under jitter, and no worse WER without jitter.

Why not just…?

Why not just train a causal model with a fixed shift and let it learn the electromechanical delay? This is the real threat. With a quiet wired link, fresh electrodes, and constant compute time, a fixed shift may do exactly as well as our system. We say so plainly. In that case the contribution narrows to the preview-curve measurement and the closed-loop study. The jitter and drift experiments settle it. A fixed shift cannot follow an arrival delay that varies from packet to packet, which the action-packet results in Wikman et al. show matters when delays are bursty. It also cannot follow an offset that drifts across blocks, as CSL-EMG_Array documents for quality.

Why not just use a chunked decoder with a few frames of look-ahead? The preview-regret result in Liu and Seiler suggests a few steps of look-ahead recover most of the acausal benefit. That is an argument for chunking, so it deserves a fair test. The difference is cost. Chunking pays its look-ahead as added latency, while electromechanical delay may supply similar look-ahead without waiting. The matched-latency comparison shows directly which one sits on the better accuracy–latency frontier.

Why not just make the model and vocoder faster? Faster compute helps both systems and composes with prediction. Prediction subtracts whatever latency remains, including wireless transmission, which cannot be engineered to zero on a wearable. Diener and Schultz also found that larger networks overfit small EMG corpora, which limits how far compute can be shaved by changing architecture alone. We report a latency breakdown so readers can see how much remains to cancel.

Why not just use offline DTW alignment as Gaddy & Klein do? We do use it, but only to build training targets. As inference, it requires the whole utterance and therefore cannot provide feedback during speech.

Proposed Experiments

The first experiment measures the preview curve on the Gaddy & Klein dataset, using the silent subset with DTW-transferred targets, and on the audible blocks of CSL-EMG_Array. We plot ASR word error rate and mel-cepstral distortion against horizon $h$. The decisive check is whether WER at $h = 40$ ms lies within a few points of WER at $h = 0$. A sharp rise immediately after $h = 0$ would falsify the free-preview premise, and we would stop the predictive line there.

The second experiment injects transmission delay drawn from Gilbert-Elliott and M/M/1 models, as used by Wikman et al., plus measured compute jitter on an embedded processor. We compare the multi-horizon system with the fixed-offset, chunked, and naive baselines at matched mean perceived latency.

The third experiment runs across CSL-EMG_Array evaluation blocks 3, 5, and 7. It tests whether online offset estimation slows the documented quality decay beyond what running normalization alone achieves.

The fourth experiment is a within-subject closed-loop study with newly recorded speakers on an EMG-UKA-style protocol, comparing four silent conditions: no feedback, Herff-style open-loop playback, our system's feedback artificially delayed to ~80 ms, and predictive near-zero-latency feedback. The outcomes are offline decoder WER on the recorded EMG, utterance duration relative to audible speech, and EMG spectral distance to the speaker's audible recordings. The decisive test is whether near-zero-latency predictive feedback beats the delayed-feedback condition. That comparison isolates timing from content.

The duration measure also bears on the conflict between reported hyperarticulation in Kashiwagi et al. and hypoarticulation in Beeson et al.. If synchronous feedback pulls both duration and spectra toward audible values, it suggests both distortions are responses to missing feedback.