Learning to Invert the Vocal Tract from Unlabeled Speech with a Differentiable Tube Model

Measuring how the tongue, lips and jaw move during speech requires sensors glued to a speaker's mouth. The articulography datasets built this way hold only 5 to 8 hours of speech from a handful of people. As a result, every current system that recovers articulator motion from audio ("acoustic-to-articulatory inversion") is a supervised regressor trained on that small pool, and it degrades on new speakers. We propose to train the inversion model mostly without articulatory labels. A recurrent network reads speech and outputs a time-varying vocal tract shape. A differentiable acoustic simulator then turns that shape back into sound, and the mismatch with the input audio is the training signal. The simulator is the classic Kelly-Lochbaum tube model, a chain of short cylinders whose reflections at each junction produce the vocal tract's resonances. Because the loop only needs audio, the model can train on hundreds of hours of ordinary speech. The physics restricts what the network is allowed to "explain" the audio with.

Two further ingredients carry the method. The first is a periodic loss: a distance that treats angles as points on a circle, so that 359° and 1° count as close. It supervises the glottal excitation phase, which a magnitude spectrogram cannot see and which gated recurrent networks track poorly. The second is a recurrent state built like a damped mass–spring system, so that inferred articulators move with inertia. The payoff we test goes beyond inversion accuracy. Co-articulation, where each sound is shaped by its neighbours, is the reason speech recognizers use context-dependent phone units. If the inferred trajectories capture real articulatory motion, they should supply a continuous representation of co-articulation. Such a representation could replace some of the expensive categorical context modelling in hybrid recognizers.

Core Claim

A recurrent encoder with physically damped articulatory state, trained by analysis-by-synthesis through a differentiable Kelly-Lochbaum tube, can recover articulatory trajectories from unlabeled speech. The encoder is trained with magnitude spectral losses plus anti-wrapping phase losses on the excitation. Adding a few minutes of EMA per speaker as anchoring, these trajectories should correlate with held-out speakers' articulography as well as or better than the best purely supervised inverter. EMA, electromagnetic articulography, is the sensor-based recording of fleshpoint positions on the articulators. The trajectories should also be useful as an auxiliary target in a factored context-dependent recognizer.

The new element is the closed loop: a neural inverter trained through a physical acoustic simulator on speech without articulatory labels. The closest pieces exist separately. The Kelly-Lochbaum inversion of Estimation studies of vocal tract shape trajectory using a variable length and lossy kelly-lochbaum model builds a codebook of 189,280 tube shapes and searches it with dynamic programming over formants. That approach is not differentiable and cannot be trained with a network. Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP goes only in the forward direction, from EMA to audio, and uses a learned filter rather than a physical tube. Speaker-Independent Acoustic-to-Articulatory Speech Inversion goes only in the inverse direction, as supervised regression. Its authors note that the resynthesis quality of its outputs stayed poor because nothing enforced acoustic consistency.

Why each ingredient is where it is

We should be precise about where phase matters, because a reviewer will object on exactly this point. A lossy Kelly-Lochbaum tube driven at the glottis is an all-pole, minimum-phase filter. Its phase response is fully determined by its magnitude response. A phase loss therefore adds nothing about the tract shape, and magnitude losses over several spectral resolutions should suffice for the filter.

Phase does carry information in the excitation. This covers the timing of glottal pulses, the instantaneous frequency of the voice source, and the alignment between synthesized and real waveforms within each frame. Magnitude-only training leaves these free. The DDSP articulatory vocoder paper reports muffled output and has to add adversarial discriminators to compensate. We therefore use the periodic losses of Low-Latency Neural Speech Phase Prediction based on Parallel Estimation Architecture and Anti-Wrapping Losses on the source side. That paper defines the anti-wrapping function $f_{AW}(x)=|x-2\pi\,\mathrm{round}(x/2\pi)|$ and applies it to instantaneous phase, group delay and instantaneous angular frequency. We borrow that loss rather than redesign it. We also borrow the cosine phase-difference losses of APNet.

The encoder does not regress F0 as an unbounded number. It carries an explicit glottal phase variable $\phi_t$ on the circle, advanced each sample by a predicted angular frequency and wrapped. This addresses the known tendency of gated recurrent networks to drift when asked to hold an oscillation.

The recurrent network is a deliberate choice, not a default. Articulators have mass. We parameterize a small set of articulatory controls $a_t$, drawn from a low-dimensional area-function model, as the positions of damped second-order systems. The recurrent cell predicts target positions and stiffness, and the state integrates them. Instantaneous jumps between phone targets become impossible, which is the physical origin of co-articulation. This also constrains the many-to-one ambiguity of inversion, where different tract shapes can produce the same spectrum. The prior is structural, not a penalty tuned after the fact.

Research Program

The simulator is a Kelly-Lochbaum tube of about 17 sections at 16 kHz, with lossy lip radiation and a variable-length lip section, following the 2010 model. It is written in JAX. Reflection coefficients are computed from areas through a clipped transform that keeps their magnitude below one, so the recursion stays stable. Areas are updated at frame rate and interpolated to sample rate.

Backpropagation through long rollouts is the known weak point of differentiable simulation. We therefore train on windows of roughly 200 ms, truncating gradients at window boundaries while carrying state forward. We will measure gradient norms against window length before scaling up.

The source is a parametric glottal pulse driven by $\phi_t$ and mixed with noise for unvoiced sounds. Turbulent fricative generation is not simulated. It is modelled as filtered noise injected at a learned constriction point, which is an explicit simplification.

Training combines three terms:

$$\mathcal{L} = \mathcal{L}_{\text{mag}} + \lambda_{\phi}\,\big(\mathcal{L}_{AW}^{\text{IP}} + \mathcal{L}_{AW}^{\text{GD}} + \mathcal{L}_{AW}^{\text{IAF}}\big) + \lambda_{\text{EMA}}\,\mathcal{L}_{\text{EMA}}.$$

Here $\mathcal{L}_{\text{mag}}$ is a multi-resolution log-magnitude spectral loss between real and resynthesized audio. The $\mathcal{L}_{AW}$ terms are anti-wrapping losses on instantaneous phase, group delay and instantaneous angular frequency of the output. $\mathcal{L}_{\text{EMA}}$ is an MSE between EMA sensor positions and a small learned linear map from $a_t$. This last term is active only on the few labeled minutes.

The encoder reads self-supervised speech features, as in the speaker-independent inversion work. It is pretrained on LibriSpeech 960h and VCTK with $\lambda_{\text{EMA}}=0$, then anchored with small amounts of MOCHA-TIMIT and HPRC.

The downstream study takes the factored hybrid recognizer of Competitive and Resource Efficient Factored Hybrid HMM Systems are Simpler Than You Think. That system regularizes a Conformer with auxiliary left, centre and right phone losses. We add an auxiliary regression onto our inferred trajectories. The specific hypothesis comes from Context-Dependent Acoustic Modeling Without Explicit Phone Clustering. That paper found that the symmetric factorization, which treats left and right context as independent given the centre phone, underperforms, while the better forward factorization is slow. A continuous trajectory couples left and right context by construction. If it does, the symmetric model conditioned on it should close much of the gap to the forward one without the forward model's decoding cost.

Baselines

The naive reference for inversion is the per-articulator mean trajectory per speaker, which fixes the floor for correlation. A slightly stronger naive reference is the non-differentiable codebook-plus-dynamic-programming inversion of the 2010 Kelly-Lochbaum paper, run on our tube model. It separates the value of gradient training from the value of the tube itself. No quantitative RMSE or correlation is reported on Lacuna for that method, so we will measure it ourselves.

The strongest existing competitor is Speaker-Independent Acoustic-to-Articulatory Speech Inversion: a bidirectional GRU with chunked autoregression, HuBERT features, adversarial and multi-task training, and speaker embeddings. It reports a Pearson correlation of 0.784 on the combined EMA datasets and is the only cited system evaluated on unseen speakers, which is our target setting. We retrain it with the same encoder features, the same held-out speaker splits and the same hyperparameter search budget. It gets two conditions: its full supervised data, and the same reduced label budget we use. Beating it means exceeding its correlation on held-out speakers at equal labels. Matching it with an order of magnitude fewer labels would also count as a clear win.

Acoustic-to-Articulatory Inversion with Deep Autoregressive Articulatory-WaveNet reports 1.08 mm average RMSE and 0.82 correlation on the EMA-MAE L1 subset. That result is speaker-dependent, so we use it only as a reference ceiling for the speaker-dependent condition, not as a head-to-head comparison.

The most dangerous baseline is a learned-vocoder loop. We close analysis-by-synthesis through the DDSP articulatory vocoder of Fast, High-Quality and Parameter-Efficient Articulatory Synthesis using Differentiable DSP instead of the physical tube, using identical data, encoder and losses. That vocoder reports 6.67% transcription WER and MOS 3.74 on MNGU0. Our system beats this baseline only if it matches or exceeds it on held-out EMA correlation. Resynthesis quality alone does not count.

Three ablations isolate the key ingredients. The first removes the anti-wrapping terms and keeps magnitude loss plus an F0 MSE. The second replaces the damped articulatory state with an unconstrained GRU output. The third sets $\lambda_{\text{EMA}}=0$ throughout and fits the linear map to EMA only at evaluation time.

For recognition, the references are the Switchboard Hub5'00 numbers of Raissi et al. (2020). CART is 13.9% WER at a real-time factor of 0.5. The forward triphone factorization is 13.6% at a real-time factor of 13.0. For the 2023 from-scratch factored system, the file gives only a relative improvement ("up to 9%"). We found no absolute WER on Lacuna for it, and we will reproduce it as our base. As a control, we add an auxiliary target of the same dimensionality taken from a random projection of the encoder's own features. This rules out a generic multi-task regularization effect.

Why not just…?

Why not just add a resynthesis loss through a learned vocoder? This is the real threat, and it is why the learned-vocoder loop is a full baseline rather than a footnote. A learned vocoder trained on one speaker's EMA, such as MNGU0, can be driven by trajectories that sound right but lie outside physiology. Nothing stops the encoder from exploiting regions the vocoder never saw in training. The tube cannot be fooled in that way, though it has its own failure mode: a vowel spectrum can be matched by a wrong but physically valid shape. The head-to-head on held-out speakers' EMA settles it. If the learned vocoder loop matches the tube on correlation, the physics is not earning its keep. In that case we would report the result as a negative finding about physical priors.

Why not just scale supervised inversion with better self-supervised features? The speaker-independent inverter already uses HuBERT and reaches 0.784. Its gains came from architecture on top of a fixed 5 to 8 hours of labels, and its resynthesis stayed poor because acoustics never entered the loss. Our label-budget sweep tests this directly. It trains both systems at 100%, 25% and 5% of the EMA data. If the supervised curve stays flat as labels shrink, the case for unlabeled training weakens.

Why not just use magnitude spectral losses and skip the periodic loss? For the tract filter we agree, for the minimum-phase reason given above, and we apply no phase loss to it. The open question is whether an uncontrolled excitation phase corrupts the tract estimates. With magnitude-only training, source errors could be absorbed by distorting the tube, which is the muffled-output symptom the DDSP vocoder paper had to patch with adversarial losses. The first ablation answers this. If removing the anti-wrapping terms leaves EMA correlation and formant error unchanged, the periodic loss is a resynthesis-quality detail, not a scientific ingredient, and we will say so.

Why not just use a Transformer or Conformer encoder? Nothing prevents an attention encoder from producing the trajectory targets. The claim concerns the state that integrates them. The damped second-order state and the wrapped glottal phase are recurrences by definition, whatever network feeds them. We will run a Conformer front end feeding the same integrator. If it matches the recurrent cell, the conclusion is that the inertial state matters and the gating does not.

Proposed Experiments

The inversion evaluation uses MOCHA-TIMIT and HPRC with leave-speakers-out splits, reporting per-articulator Pearson correlation and RMSE. A formant-tracking check compares the tube's resonances against formants measured by linear prediction on held-out audio. Resynthesis is measured by mel-cepstral distortion on CMU ARCTIC, the same zero-ground-truth setting as the speaker-independent paper. The label-budget sweep described above is the central curve.

The first decisive check is whether the unlabeled pretraining on LibriSpeech and VCTK moves held-out correlation at low label budgets. If it does not, the core claim fails regardless of resynthesis quality. The second is the learned-vocoder comparison. The third is a sanity test on vowel-to-vowel transitions (/a/–/u/–/i/): the smoothness the 2010 dynamic-programming method achieved should emerge here from the damped state without search.

The recognition study runs on Switchboard 300h (Hub5'00 and Hub5'01) and LibriSpeech 960h. Its decisive comparison is the symmetric factorization with and without the trajectory auxiliary. We ask whether the gap to the forward factorization shrinks, and at what real-time factor, relative to the random-projection control. A null result here would still be informative. It would suggest that the categorical context in hybrid recognizers captures something the continuous trajectory does not, rather than merely approximating articulator inertia.