Speed Perturbation as a Geometric Teacher for Articulatory Inversion of Disordered Speech

Speeding up or slowing down recordings is the default augmentation in speech recognition, and for disordered speech it helps little. It stretches the whole utterance by one factor, while dysarthria (speech impaired by weak or poorly coordinated muscles, as in ALS, cerebral palsy or Parkinson's disease) produces local, uneven distortions in how the tongue, lips and jaw move. This note proposes a different use for the same transformation. When we tempo-perturb an utterance, we know exactly how the hidden articulator movements were warped in time. That makes it a free, label-free constraint for a model that infers articulator positions from audio, a task called acoustic-to-articulatory inversion. We train the inversion model to predict a probability map over the mid-sagittal vocal tract plane for each articulator sensor, rather than a single coordinate. We then require its maps on perturbed audio to match the correspondingly warped maps of the original. Because this constraint needs no articulatory labels, it can be applied to unlabelled dysarthric speech. That is the domain where current inversion models break.

The opportunity exists because two findings in the literature point the same way. Articulatory features estimated from audio cut word error rate (WER) for elderly and dysarthric speech even on top of speed perturbation and speaker adaptation (Hu et al.). Yet inversion models that regress point trajectories fail badly outside their training conditions: error rises from 0.49–0.71 mm to 2.96–3.10 mm, and correlation drops from about 0.9 to about 0.3, when moving from sensor-encumbered to natural speech (Fang). If the articulatory signal is valuable but the inverter is brittle, the obvious step is to make the inverter transfer. A second step follows: expose its uncertainty to the recogniser instead of a falsely confident point.

Core Claim

An inversion model has three properties. It outputs spatial heatmaps (2D probability maps whose peak marks an articulator location). It is trained with supervised targets on healthy speakers, together with equivariance under known audio transformations on unlabelled dysarthric speakers. And it projects through a low-dimensional articulator shape model. Such a model will track articulators of dysarthric talkers more accurately than point-regression inversion. Its features, which include per-articulator spread as well as location, will give larger WER reductions than point-trajectory features under the same recogniser.

What is new is the pairing of three choices. The first is to use acoustic perturbations as geometric self-supervision for inversion rather than as recogniser augmentation. The second is the heatmap formulation of inversion. The third is a test on TORGO, which records electromagnetic articulography (EMA, small coils glued to the tongue and lips whose 3D positions are tracked) for dysarthric speakers themselves. That last point lets us measure inversion transfer directly instead of inferring it from downstream WER.

Research Program

Inversion model. The inversion network reads log-mel frames. For each EMA coil it outputs a low-resolution heatmap over a speaker-normalised mid-sagittal grid. Supervised targets are Gaussians centred on the measured coil positions, the standard formulation from pose and landmark work (Pfister et al.; Bulat & Tzimiropoulos). Subpixel offsets follow Subpixel Heatmap Regression to limit grid quantisation. We borrow heatmaps for a reason the landmark literature gives explicitly. Mean-squared coordinate regression averages over plausible locations, and under ambiguity that average can land in anatomically impossible space (Rectified Wing Loss). Imprecise dysarthric articulation is exactly the case where several tongue positions are consistent with the same distorted spectrum.

Equivariance objective. For unlabelled audio $x$, we apply tempo perturbation $\tau_r$ at rate $r$, which changes duration but not pitch. We then require:

$$\mathcal{L}_{eq} = \sum_{t}\sum_{k} \mathrm{KL}\big(\,\mathrm{sg}[\mathcal{W}_r H(x)]_{t,k} \;\|\; H(\tau_r x)_{t,k}\,\big)$$

Here $H(\cdot)_{t,k}$ is the predicted heatmap for coil $k$ at frame $t$, $\mathcal{W}_r$ resamples the heatmap sequence in time by the same factor $r$, and $\mathrm{sg}$ stops gradients through the target branch. Two further transformations follow the same pattern:

  • Resampling-based speed perturbation. This also scales formants, roughly imitating a shorter or longer vocal tract. We treat it as a transformation under which positions in the normalised grid should be unchanged.
  • Local piecewise time warps. We add random non-uniform time warps. Their exact warp is also known, which partly addresses the complaint that uniform warps do not resemble dysarthric timing.

Preventing collapse. Equivariance alone is satisfied by a constant output, which is the degenerate solution that lift-project loops are known to fall into. We prevent it in two ways. First, the supervised loss on healthy EMA stays on throughout training. Second, each frame's heatmap peaks are projected onto a low-dimensional articulator shape model, fitted by principal components on control-speaker EMA configurations, and re-rendered as heatmaps. A consistency loss between the raw and re-rendered maps keeps predictions on the manifold of physically reachable vocal tract shapes. This lift-project-compare loop is the mechanism used for 3D pose and articulated shape by Sosa et al., Gupta et al. and Cao. We reuse it with an articulatory template in place of a camera and a body.

Curriculum, in a secondary role. Curriculum enters as a modest, testable ingredient rather than the headline. Heatmap target width $\sigma$ starts wide and shrinks over training, so early training learns coarse vocal tract layout before fine localisation. The perturbation magnitude, meaning the range of $r$ and warp strength, ramps from low to high. This follows the affinity/diversity logic of Gontijo-Lopes et al.: start with augmented samples close to the clean distribution, then add diversity. It also follows the progressive-difficulty schedule used for phoneme negatives in DyPCL. We will measure affinity of the perturbed audio with a clean-trained inverter to set the schedule end points rather than hand-tune them. If the curriculum does not beat a fixed $\sigma$ and fixed magnitude, we drop it from the method.

Recogniser interface. Per frame and per coil, the recogniser receives the heatmap expectation (location), its covariance (spread) and its entropy. These are fused with acoustic features through hidden-layer fusion, as in Hu et al.. The hypothesis that uncertainty itself is informative is tested by withholding the spread and entropy channels.

Baselines

The naive reference is the recogniser alone, a Conformer trained on acoustic features with standard speed perturbation and SpecAugment, and no articulatory input. Every other system adds to this same backbone, data split and training budget. A gain over it measures whether articulatory information contributes at all.

The strongest directly comparable competitor is point-trajectory inversion feeding the recogniser, as in Hu et al.. There a BLSTM predicts articulatory features from audio, and they are fused into the recogniser. That paper reports a 2.07% absolute (22.72% relative) WER reduction on TORGO and 4.75% absolute on DementiaBank Pitt, on top of speed perturbation and speaker adaptation. Their inverter was trained on ultrasound data from TaL, while ours is trained on EMA. So we reimplement their recipe on TORGO control EMA with our backbone, the same inverter capacity and the same hyperparameter search budget. We also report the published deltas for context. Beating it means two things on held-out dysarthric TORGO speakers: lower inversion RMSE against measured EMA, and a larger absolute WER reduction over the shared naive reference than the reimplemented point-trajectory features.

The strongest acoustic-only competitor is DyPCL. It is a HuBERT-based recogniser with curriculum phoneme-level contrastive learning, reporting a 22.10% relative WER reduction on UASpeech and very-low-intelligibility WER falling from 58.49% to 49.45%. We do not need to beat it outright to make our point. The claim is that articulatory features are complementary. So we run DyPCL as published on UASpeech, then DyPCL with our fused features. Success means a significant further reduction, concentrated in the low-intelligibility groups. Personalized adversarial augmentation reports up to 0.91% absolute on TORGO and serves as the learned-augmentation reference point.

Three ablations isolate the ingredients:

  • Heatmaps versus coordinates. We replace the heatmap head with a coordinate head that also predicts a Gaussian variance, trained by negative log-likelihood. This separates "heatmaps" from merely "uncertainty-aware outputs."
  • Equivariance. We remove the equivariance loss on unlabelled dysarthric audio, isolating the self-supervision.
  • Shape model and curriculum. We remove the shape-model projection, and separately replace the curriculum with fixed $\sigma$ and fixed magnitude.

Why not just…?

Why not just use stronger acoustic augmentation (heavier SpecAugment, pitch shifting, randomised speed schedules)? These change the channel and timing of the signal but not the production mechanics behind it. The file's evidence is that this ceiling is real. Tempo perturbation gave 31.79% WER against 28.86% for learned GAN-based adaptation on UASpeech (Jin et al.). Articulatory features still helped on top of speed perturbation (Hu et al.). The naive reference in our design already includes the strongest acoustic augmentation recipe we can tune, so this alternative is measured rather than assumed away.

Why not just synthesise dysarthric audio with TTS, voice conversion or articulatory synthesis? Zero-shot TTS degraded with severe dysarthric reference audio and was unstable (Speech Synthesis augmentation). Adding VocalTractLab articulatory synthesis to real German data lowered overall phoneme recall and F1 (Articulatory Synthesis). Our approach never trains on synthetic audio. It only uses label-free transformations of real dysarthric recordings, so it avoids the naturalness gap these results expose.

Why not just regress coordinates with a predicted variance? This is the most serious threat. It is cheaper than heatmaps and also yields uncertainty. Its weakness is that it is unimodal. When two tongue configurations explain a sound, it reports their midpoint with a wide variance, which is the averaging failure described in Rectified Wing Loss. Whether that matters for speech is an empirical question we have no prior evidence on. The coordinate-plus-variance ablation, trained with the identical equivariance and shape-model losses, settles it. If it matches heatmaps on dysarthric EMA RMSE and downstream WER, we adopt it and the contribution becomes the self-supervision alone.

Why not just fine-tune a large self-supervised speech encoder and skip articulators? DyPCL shows how far acoustic representations can go. The concern is that Hernandez et al. found UASpeech and TORGO classifiers reach 96–100% accuracy on silence alone, so acoustic models can learn recording-setup shortcuts (Hernandez et al.). Physically constrained articulator positions should carry little of that channel information. The DyPCL-plus-features experiment and the silence check below test this jointly.

Proposed Experiments

Inversion transfer. Train on TORGO control speakers' EMA, using dysarthric speakers' audio only through the equivariance loss with their EMA held out. Then measure RMSE and Pearson correlation against the dysarthric speakers' measured EMA, per speaker and per severity. The decisive outcome is a narrowing of the healthy-to-dysarthric error gap relative to the point-regression baseline and to the no-equivariance ablation. Fang's 4–6 fold out-of-domain degradation is the failure pattern we expect the baseline to show. Where the EMA-synchronised versus stand-alone recordings are accessible, we repeat the same protocol on that sensor mismatch.

Recognition. We measure WER on TORGO with speaker-independent splits. We also measure WER on UASpeech, which has no articulatory data, using the TORGO-trained inverter. There we report by intelligibility group, since the very-low group is where both DyPCL and the adversarial methods concentrate their gains. A further run on DementiaBank Pitt allows comparison with Hu et al.'s elderly-speech result. Significance is assessed by matched-pair tests over utterances.

Shortcut check. Following Hernandez et al., we train a dysarthric-versus-control classifier on silence segments only, once using acoustic features and once using our articulatory features. If the inferred articulators are physically grounded, the articulatory classifier should fall toward chance on silence while remaining informative on speech. If it does not, the features are leaking channel information, and the WER gains must be interpreted with that caveat.

Uncertainty check. We correlate per-frame heatmap entropy with intelligibility rating and with frame-level recognition errors. Withholding the spread and entropy channels from the recogniser should cost WER mainly for low-intelligibility speakers. If it costs nothing, the heatmap formulation's case rests on inversion accuracy alone.