Sealing the Airway: Contact-Exact Acoustic-to-Articulatory Inversion of the Whole Vocal Tract

Models that animate human bodies from text or music often let the feet glide across the floor while they supposedly carry the body's weight. This happens because the networks predict smooth joint positions and have no notion of contact. This note argues that speech models have the same defect in a place nobody has measured. Acoustic-to-articulatory inversion means predicting the shape of the vocal tract from the sound of speech. Its most complete versions now recover the whole tract from real-time MRI, from lips to vocal folds. These models report average contour errors of about 1.65 mm. That figure is close to the MRI pixel size, and it hides the frames where errors matter most. During a /t/, /k/, /p/ or /n/, the tongue, lips or velum must fully close the airway against the palate, the other lip, or the pharyngeal wall. A 1.5 mm error at that moment leaves a false air gap, which acoustically turns a stop into something like a fricative. It can also push the tongue through the bony hard palate. Neither failure shows up in an average error.

We propose to measure this "articulatory foot sliding" and then remove it by construction. The model will not predict two airway walls independently. It will predict one wall plus a non-negative opening between the walls, together with a contact gate that can set that opening exactly to zero. Under this scheme penetration cannot occur and a closure is exactly sealed. Unlike the foot-locking fix used in motion synthesis, the point of contact can still slide along the palate, as real tongues do. A learned, differentiable forward model maps vocal-tract geometry to sound and acts as a fast stand-in for acoustic simulation. It is used in two ways: to ask whether the geometric fix changes the predicted acoustics, and as an optional training signal. A second, independent acoustic judge checks that the inversion is not simply fooling that learned model.

Core Claim

Whole-tract inversion models are judged by spatially averaged contour distance. That measure is blind to the errors at closures that decide what sound is produced. The fix that transfers from motion synthesis is to make contact an explicit, discrete state of the representation, so that sealing and non-penetration are guaranteed rather than encouraged by a loss. The specific motion-synthesis constraint, zero velocity during contact, does not transfer, because tongue contact rolls and slides. The concrete claim is that an aperture-plus-contact-gate parameterization will cut penetration and false-gap rates during stop and nasal closures far below those of an unconstrained bidirectional LSTM inverter. It should do this without raising overall contour error, and without introducing false closures during fricatives and approximants, where articulators come close but must not seal. Post-hoc snapping and soft contact losses are the two cheaper fixes, and we expect the hard parameterization to beat both on that last point.

Where this sits in prior work

The regime comes from the whole-tract imaging literature, which argued that sparse sensors miss the pharynx, velum and larynx. A study of emotional speech articulation using a fast magnetic resonance imaging technique showed large pharyngeal and laryngeal reconfigurations that point tracking cannot see. The corpora we build on are ArtSpeechMRIfr, which offers 55 fps midsagittal MRI, 3D static MRI and reconstructed dentition, and USC-TIMIT, which offers 23.18 fps MRI for four speakers.

The direct baselines are two inversion systems. Reconstruction of the Complete Vocal Tract Contour Through Acoustic to Articulatory Inversion Using Real-Time MRI Data regresses eight articulator contours with bidirectional LSTMs. Speaker Dependent Acoustic-to-Articulatory Inversion Using Real-Time MRI of the Vocal Tract regresses raw MRI pixels. Both optimize distance-type losses and neither reports anything about contact.

The closest work is Optical Flow Guided Tongue Trajectory Generation for Diffusion-based Acoustic to Articulatory Inversion. It adds an optical-flow trajectory loss to a diffusion model that predicts ultrasound tongue images. That makes tongue motion smoother and more consistent. It has no representation of the palate, no contact state, and no guarantee against gaps or penetration, which is the part we add.

Two papers supply the method. Combining Recurrent Neural Networks and Adversarial Training for Human Motion Synthesis and Control showed that predicting explicit binary foot-contact labels alongside motion removes skating. We reuse the explicit contact state but replace its zero-velocity rule. Articulatory Characteristics of Icelandic Voiced Fricative Lenition and the Italian affricate study Lexical stress affects lenition both document gradient, sliding, near-contact gestures, and these constrain the design. A rule that freezes the articulator on contact would be wrong for speech, and a threshold-based snap would wrongly seal lenited consonants.

Learning Incompressible Fluid Dynamics from Scratch supplies the design principle. It enforces incompressibility by predicting a quantity whose curl is the velocity, so the constraint holds for any network output. Our aperture parameterization does the same for non-penetration.

On the acoustic side, Speaker Dependent Articulatory-to-Acoustic Mapping Using Real-Time MRI of the Vocal Tract is the existence proof for a learned geometry-to-acoustics model, reaching 4.02 dB mel-cepstral distortion with a CNN-LSTM. We repurpose a model of this kind as a differentiable surrogate. The surrogate literature, including Fast acoustic scattering using convolutional neural networks, shows such models are fast. It also warns that they degrade outside their training geometries, which matters here because penetrating or half-closed tracts never appear in real data.

Research Program

Airway representation. We first fix a per-speaker coordinate system of grid lines running from the lips to the glottis, the classic midsagittal-distance layout. It is anchored to the speaker's rigid structures: the hard palate and teeth taken from the ArtSpeechMRIfr dental reconstruction, and the upper lip as reference for the labial region. In each region, one wall is the reference wall: the palate, the posterior pharyngeal wall, or the upper lip. Reference walls that move, such as the pharyngeal wall and the velum, are predicted by an ordinary contour regressor; they are smooth and low-variance. The opposing active wall is then defined by its distance from the reference wall along each grid line.

Aperture and contact gate. For grid line $k$ at frame $t$, the network outputs an unconstrained opening score $a_k(t)$ and a contact probability $c_k(t) \in [0,1]$. The aperture, meaning the open distance between the two walls, is

$$d_k(t) = \big(1 - \hat c_k(t)\big)\,\mathrm{softplus}\big(a_k(t)\big),$$

where $\hat c_k(t)$ is the contact gate. During training it is relaxed with a straight-through estimator; at inference it is hard-thresholded. The active wall point is the reference wall point plus $d_k(t)$ times the inward normal of the reference wall. This gives two guarantees for any network output: $d_k \ge 0$, so there is no penetration, and $d_k = 0$ exactly when the gate fires, so the seal is exact.

Sliding is allowed. Contact is a property of individual grid lines, not of the whole articulator. The set of sealed grid lines can therefore shift forward or backward over time, which lets the contact point slide. No velocity constraint is imposed anywhere.

Contact labels. Contact supervision comes from the MRI contours. A grid line counts as sealed when the measured walls are within one pixel of each other, and this is cross-checked against forced-aligned closure intervals of stops and nasals. A 2D midsagittal seal is not a true 3D seal; laterals let air escape at the sides, for example. We therefore restrict the "must seal" evaluation to stops and nasals, whose closures are midsagittal.

Encoder. The network that turns audio into these outputs is the same bidirectional LSTM used by Azzouz et al. This isolates the representation as the only change.

Acoustic surrogate. We train a CNN-LSTM forward model from geometry, rasterized from contours, to mel spectra, following Csapó et al., on the same speaker's training split. It serves two roles. First, it gives an acoustic-consequence metric: we resynthesize from predicted geometry and from ground-truth geometry and compare the two. Second, in an ablation, it provides an analysis-by-synthesis loss that pushes the inverter toward geometries that reproduce the input audio.

Guarding against surrogate exploitation. A learned surrogate may reward shapes it has never seen. Every acoustic claim is therefore re-checked with an independent physics judge: a classical 1D tube model, driven by area functions converted from midsagittal distances and calibrated per speaker on the ArtSpeechMRIfr 3D static scans. Its absolute accuracy is limited, but it is enough to tell a sealed tract from an open one.

Cross-speaker transfer. As a smaller secondary test, we ask whether the aperture representation survives transfer between the two ArtSpeechMRIfr speakers. A model trained on one speaker is fine-tuned on a few minutes of the other, with the new speaker's palate as the anchor. Because a zero aperture is zero for any anatomy, closures should transfer even when spatial warping would break them.

Baselines

Naive reference points. The first is a phone-conditioned mean shape: for every frame, the average training contour for the aligned phone. It shows how much closure behavior any model gets just from knowing the phone identity. The second is a re-implementation of the Azzouz et al. articulator-by-articulator bidirectional LSTM. That paper reports 1.65 mm average RMSE on its single-speaker French data, with 1.67 mm mean and 1.45 mm median for the wider-context variant. On image-based inversion, Csapó et al. report an LSTM average NMSE of 0.0036 and SSIM of 0.78 on USC-TIMIT. We found no reported result on Lacuna for any contact, penetration or closure metric on any of these corpora, so those columns will be established here for the first time. The LSTM is the strongest existing method on this exact question: it is the best published whole-tract contour inverter, and its outputs carry the contour accuracy that any contact fix must not degrade.

Strongest cheap fix. We apply signed-distance snapping to the baseline's outputs. Any active-wall point inside the reference wall is projected onto it, and any grid line with an aperture below a threshold is closed. The threshold is tuned on validation data to maximize closure F1. This is the speech analogue of inverse-kinematics foot locking.

Soft-loss ablation. We use the identical contact head and grid-line representation, but the aperture is unconstrained. A penalty on predicted aperture during labeled contact and a hinge penalty on negative aperture replace the hard gate. This ablation isolates the proposal's key ingredient, guarantee by construction versus encouragement by loss. A second ablation replaces the aperture rule with the Xia et al. zero-velocity rule on contact frames, to test our claim that freezing is wrong for sliding tongue contact.

Re-weighting. Contour RMSE is up-weighted near constrictions. This checks whether simply attending more to closures suffices.

Fair comparison. All models share the encoder, the acoustic features, the train, validation and test splits, an equal hyperparameter search budget of 20 configurations each, and three seeds. We report means and spreads.

What counts as winning. The proposal wins if it reduces stop and nasal false-gap and penetration rates by a margin larger than the seed spread compared with snapping and the soft loss, while keeping overall RMSE within 0.05 mm of the baseline and keeping the false-closure rate on fricatives and approximants no higher than the baseline's.

Why not just…?

Why not just snap the baseline's tongue onto the palate after the fact? This is a real threat, and it is why snapping is a full baseline. Snapping trivially removes penetration. Its weakness lies in the lenition literature cited above: fricatives, approximants and lenited stops come within a millimeter or two of the palate without sealing. Any snap threshold that catches the baseline's sloppy stop closures will also seal those near-contacts. Snapping also creates frame-to-frame jumps at the threshold crossing. The experiment settles the question by plotting stop-closure F1 against fricative false-closure rate across all snap thresholds. If some threshold matches the gated model on both, the honest conclusion is that the contribution reduces to the diagnostic metrics, and we will report it that way.

Why not just add a contact head and a zero-velocity or aperture loss, as in motion synthesis? The zero-velocity version is likely wrong for speech. Tongue contact during affricates and diphthongs slides, as the Italian affricate and Icelandic studies show, and freezing it should distort these gestures. Our ablation measures exactly that. The soft aperture-loss version is the more serious competitor. It may close stops nearly as well, and its remaining failures would be small residual gaps and occasional penetrations. These are small in millimeters, but large acoustically, because a 0.5 mm gap still leaks air. Comparing soft against hard on residual-gap rate and on the tube-model judge of sealing decides this. If the soft loss reaches the same sealing rates, the "by construction" claim fails.

Why not just use a 1D tube model as the acoustic signal and skip the learned surrogate? This is partly conceded: the 1D tube is our independent judge precisely because it is physically grounded and cannot be fooled the way a learned model can. ArtSpeechMRIfr's authors report that midsagittal-to-acoustic simplifications miss transverse width, so the tube's formants are only roughly right. The learned surrogate, trained on real paired audio, should match real spectra more closely; Csapó et al. report 4.02 dB. Its weakness is the known out-of-distribution fragility of surrogates. The ablation that adds the surrogate loss to the unconstrained baseline tests exactly this: if surrogate distortion improves while tube-judged sealing and contour accuracy worsen, the surrogate is being exploited. Our prediction is that the aperture parameterization limits such exploitation, because it keeps the inverter's outputs within the family of shapes real tracts produce.

Why not just collect more speakers? Text-to-motion models trained on large motion-capture data still slide their feet, as noted for TEMOS. MRI collection is also slow and fragile; USC-TIMIT lost 74% synchronization for one speaker. More data does not offer a guarantee that a closure is sealed.

Proposed Experiments

Primary data. The primary corpus is ArtSpeechMRIfr. Its 55 fps rate gives three to five frames per stop closure, and its dental reconstruction gives an accurate hard-palate anchor. We use both speakers, with held-out sentences. If the Azzouz et al. single-speaker data is accessible, it lets us reproduce the 1.65 mm baseline directly.

Secondary data. USC-TIMIT's four speakers test robustness. Its 23.18 fps rate is a stress condition in which many closures last only one or two frames. Speaker m1, with its documented desynchronization, is reported separately.

Metrics. Overall and constriction-region contour RMSE. Frame-level closure F1 within stop and nasal intervals. False-gap rate and median residual gap during closures. Penetration rate and depth into the palate and teeth. False-closure rate during fricatives and approximants. Contact-point drift during held closures, compared with the drift measured in ground truth rather than with zero. Acoustic consequence, measured both as surrogate mel-cepstral distortion between resynthesis from predicted and from true geometry, and as a tube-model check of whether a sealed tract occurs during each labeled closure.

First decisive check: is the problem real? If the baseline LSTM already seals more than 95% of stop-closure frames without penetration, the premise fails and we stop there.

Second decisive check: does the guarantee buy anything? The hard gate must beat both snapping across its full threshold curve and the soft-loss ablation on sealing, without raising fricative false closures.

Third decisive check: does the geometric fix matter acoustically? Improvements must show up under the physics tube judge, not only under the learned surrogate. A divergence between the two judges is itself a finding about surrogate exploitation.

Transfer check. We compare closure F1 after few-minute adaptation to the second speaker against a baseline adapted by spatial warping of the first speaker's contours.