When There Is Nothing to Attend To: Outliers from Silent Speech and Clock Drift in Quantized Speech Interfaces

A silent speech interface reads words from the movements of the mouth instead of from sound. It might combine a small ultrasound probe under the chin that images the tongue with a camera that watches the lips. To run on a wearable, such a system must be compressed to low-bit integer arithmetic, a step called quantization, and it must fuse sensors that each keep time with their own cheap crystal oscillator. This proposal starts from one observation. Two deployment conditions remove the evidence that a fusion model's attention layers are built to find. The first is silent articulation: people mouth words with smaller, slower tongue movements and different lip movements than when they speak aloud. The second is inter-device clock drift, where two sensors' sampling clocks slowly slide apart. Transformers that cannot find anything useful to attend to are known to fake a "do nothing" operation, and the extreme activation values this produces are exactly what breaks low-bit quantization. We hypothesize that silent speech and clock drift create new activation outliers that never appeared in the vocalized, perfectly synchronized data used for training and quantization calibration. If so, a compressed model that looks fine in the lab would fail specifically on the users and hardware it was built for.

The proposal has two parts. The first is a measurement: how much outlier growth and quantization damage appear when vocalized, synchronized training data is replaced by silent, drifting test data. The second is a remedy, built and tested against simpler options. We train the fusion model on abundant vocalized speech with simulated clock drift, and we give its attention heads an explicit way to output nothing. The goal is a single 4-bit model whose calibration transfers from vocalized to silent speech and from simulated to real drift. We treat speaking mode mismatch and drift as two instances of one sim-to-real gap. Vocalized, synchronized recordings play the role of the "simulator," and silent speech captured on separately clocked devices plays the role of "reality."

Core Claim

Quantizable Transformers showed a specific cause of outliers. Because softmax attention weights must sum to one, a head that should not update a token instead pushes its logits to extremes and parks attention on an uninformative token. This drives activations in the next layer to huge values. Their gated attention removes the problem: OPT-1.3B at INT8 reaches 29.95 perplexity versus 989.6 for the vanilla model. Anatomy of Massive Activations and Attention Sinks adds that sink formation depends on training conditions such as head dimension and context length. Both works study text and images from a fixed distribution. Neither asks what happens when the input distribution itself shifts so that no-op behavior becomes more frequent at test time. That is our question.

Silent speech is a strong candidate for such a shift. Silent versus Modal Multi-Speaker Speech Recognition from Ultrasound and Video found that silent utterances are longer and have a significantly smaller tongue movement area. It also found that word error rate (WER) on the TaL80 corpus rises from 39.34% to 77.79%. Silent Speech Recognition with Articulator Positions Estimated from Tongue Ultrasound and Lip Video documents tongue targets being undershot. Visual work (Visual-Only Recognition of Normal, Whispered and Silent Speech; Improving the Gap in Visual Speech Recognition Between Normal and Silent Speech Based on Metric Learning) reports exaggerated lip movement instead. A cross-modal head trained to match tongue shapes to lip shapes would therefore see weaker tongue evidence and unfamiliar lip evidence. Clock drift makes this worse. Once the streams have slid apart by a few frames, the frames that a head pairs across modalities no longer describe the same instant.

The claim has three parts, and each can be falsified on its own:

  1. In a vanilla fusion Transformer trained on vocalized, synchronized data, activation peak magnitudes and the share of attention sent to sink tokens rise on silent inputs and under drift.
  2. This rise, not the accuracy drop at full precision, predicts the extra WER lost when the model is quantized at 4 bits.
  3. Gated attention, trained with simulated drift, keeps activations bounded across both shifts. A quantizer calibrated only on vocalized, synchronized data then transfers to silent, drifting data.

Research Program

The model is a Transformer silent speech recognizer. It has an ultrasound encoder and a lip-video encoder, fused by cross-attention in both directions and trained with a CTC (connectionist temporal classification) objective. The TaL results above come from DNN-HMM (neural network plus hidden Markov model) pipelines, so this Transformer is a new build. We train it on the modal (vocalized) portions of TaL80 and TaL1 and evaluate on their silent portions. A video-only variant is trained and tested on AV Digits and the Petridis normal, whispered, and silent corpus. This variant separates the mode effect from the cross-modal effect.

Drift simulation resamples one stream relative to the other. The resampling combines a fixed start-up offset, a constant rate mismatch, and a slow random-walk component. The random walk stands in for temperature- and load-dependent wandering, of the kind Nayak et al. observed to differ across wireless links. That paper trained its echo canceller on simulated rooms but relied on real recordings to teach drift. We instead ask whether simulated drift alone suffices. To test this, we collect a small real-drift set: a few speakers reading TaL-style prompts silently and aloud, with ultrasound on its own acquisition clock and lips on an independent consumer camera. Light flashes at the start and end of each session record the true offset and skew. As far as the file shows, no public dataset captures real clock drift for articulatory sensors, so this set is a contribution in its own right.

The analysis records three quantities at every layer. These are the maximum absolute activation (the "infinity norm" used by Bondarenko et al.), kurtosis, and the fraction of attention mass on low-information tokens. We cross these with speaking mode, drift magnitude, and bit width (W8A8, W4A8, W4A4, where W is weight bits and A is activation bits). The remedy variants are gated attention, clipped softmax, and DynamicTanh normalization. The file reports that DynamicTanh cut spike magnitude from 3,818 to 153 in text models. Each variant is trained with and without drift augmentation.

Baselines

The naive reference point is the vanilla fusion Transformer, trained on synchronized modal data. It is quantized with standard per-tensor min-max calibration on held-out modal synchronized utterances and tested on silent and drifted inputs. Its FP16 silent WER sets the full-precision ceiling. The gap between its FP16 and quantized WER is the damage we are trying to explain. We will not claim a comparison to the 77.79% TaL80 silent WER of Ribeiro et al., because that number comes from a different pipeline and feature set. We will report it only as context, next to the 23.06% relative WER gain Beeson et al. obtained from landmark features.

The strongest post-hoc competitor is Outlier Suppression+. It shifts and scales each channel and folds the inverse into neighboring weights, and it reports a 15.5% GLUE improvement at 4-bit BERT-base over prior methods. It is the best-documented way to tame channel outliers without retraining. It runs on the same frozen vanilla model with the same calibration set. We will also run the token-wise dynamic activation quantization of ZeroQuant, which recomputes ranges per token at runtime and so adapts to distribution shift by design. For quantized silent speech recognition there is no reported result on Lacuna for any method. The strongest reported number for the architectural remedy is Bondarenko et al.'s gated-attention OPT-1.3B INT8 perplexity of 29.95 against 989.6 for the vanilla model, in text.

The key-ingredient ablations separate the architecture from the augmentation. They compare gated attention without drift augmentation, drift augmentation without gating, and both. All models share one encoder size, training budget, and hyperparameter search budget, and every quantizer sees the same number of calibration utterances.

We beat the baselines if two conditions hold. First, the gated, drift-augmented model's 4-bit (W4A8) silent WER on TaL80 must lie closer to its own FP16 WER than OS+ and ZeroQuant manage on the vanilla model, and this must hold under both simulated and real drift. Second, its own FP16 silent WER must be no worse than the vanilla model's.

Why not just…?

Why not just resynchronize the streams before the model? If device timestamps or a cross-correlation estimate can undo drift, the model never sees it. This is a real threat to the drift half of the proposal. We will run an oracle using the true offset and skew from the flash markers, plus a practical estimator, both feeding the vanilla model. If oracle resynchronization plus OS+ matches our method under drift, we will report that drift is a preprocessing problem, and the contribution narrows to speaking mode. Resynchronization cannot remove mode-induced outliers, and the video-only experiments test that effect with no drift at all.

Why not just calibrate the quantizer on the user's own silent speech? Calibration needs only unlabeled activations, so a user could supply a few minutes of mouthing. This may well work, so we measure it directly by calibrating on 10, 50, and 200 unlabeled silent TaL80 utterances. If a small unlabeled set closes the gap, the finding becomes a cheap deployment recipe: "calibrate on silent speech." The remedy's value then reduces to cases where per-user collection is impractical, or where outliers vary within a session, for example as drift accumulates. The long-session real-drift recordings test that second case.

Why not just use dynamic per-token activation ranges? ZeroQuant's per-token scaling tracks shifts that appear as whole-token spikes. However, the text reports its own attention inputs needing FP16 fallback at scale. SVDQuant shows that at 4 bits, the outlier burden cannot simply be moved into the weights. Channel-structured outliers of the kind Outlier Suppression documents are also not solved by per-token ranges. We include ZeroQuant as a full baseline at W4A4, where this limitation should show if it matters.

Why not just keep attention inputs in FP16 or stay at 8 bits? Nothing new would be needed at INT8 if the vanilla model already survives there. We report W8A8 precisely to find the bit width where the mode and drift effect begins. Alignment Collapse Under KV Cache Quantization is the relevant warning: aggregate metrics can look fine while a subpopulation fails. So every result is broken down by speaking mode and drift level rather than averaged.

Proposed Experiments

The first decisive check is diagnostic. On the vanilla model, we plot peak activation and sink fraction against speaking mode (modal, whispered, silent) and against drift magnitude. We then test whether the increase predicts quantization-induced WER loss better than full-precision WER does. If outliers do not grow under silent speech or drift, the core hypothesis is false and we stop at a negative result. That result would still be useful, since it would show that articulatory mode shift is benign for quantization.

The second check is the remedy comparison described under Baselines. It runs on TaL80 and TaL1 silent speech (with ultrasound and lips) and on AV Digits silent speech (video only). The metrics are WER, viseme error rate on AV Digits, where Kashiwagi et al. report 9.97% silent WER at full precision, and the FP16-to-quantized gap. The third check is the sim-to-real test for drift. We compare models trained on simulated drift on the real separately clocked recordings, and ask whether the ordering of methods seen under simulated drift survives. The fourth check tests the mechanism. Sun et al. find that head dimension drives sink formation, so we train vanilla models with smaller heads and test whether the mode-induced outlier growth shrinks accordingly. That would tie the effect to the no-op mechanism rather than to generic distribution shift.