Amortized Articulatory Posteriors from Unlabeled Speech

Recovering how the tongue, lips and jaw moved from a recording alone is called acoustic-to-articulatory inversion. It is a long-standing problem, and it underlies interpretable speech codecs, speech-pathology tools and silent-speech interfaces. Today it is solved in one of two unsatisfying ways. The first is to regress onto electromagnetic articulography (EMA), which means gluing sensor coils to a speaker's tongue, so only a handful of hours from a few dozen speakers exist. The second is to run gradient descent through a differentiable vocal-tract simulator separately for every clip. That removes labels, but each utterance costs thousands of optimization steps and returns a single guess. A single guess is a real flaw here, because different vocal-tract shapes can produce the same sound. Speech scientists call this acoustic equivalence, and it makes the true answer a distribution, not a point.

We propose to train one network, once, that maps any recording to a probability distribution over simulator vocal-tract trajectories. The network is trained using only unlabeled speech and a differentiable physical simulator. Instead of pretending the simulator is a faithful model of human voices, we treat its spectral mismatch as a loss with an adjustable trust level. This is "amortized inference": paying a one-time training cost so that each new input is handled in a single forward pass. We amortize a tempered, loss-based posterior over real speech, and we test whether this cuts the amount of EMA data needed to read out articulator positions. The headline question is a label-efficiency one. Can a physically grounded, uncertainty-aware inverse model reach the articulatory accuracy of EMA-supervised systems using minutes, not hours, of sensor data, and using those minutes only for a final linear readout?

Core Claim

Let $g(\theta)$ be the differentiable Pink Trombone simulator from Articulatory Synthesis of Speech and Diverse Vocal Sounds via Optimization. That paper (VocalTrax) reimplements a 2D vocal tract with 1D wave propagation in JAX and inverts clips by per-clip mel-spectrogram optimization. Here $\theta$ is a frame-by-frame trajectory of tongue, constriction, lip and glottal-tenseness parameters, with pitch fixed from a pitch tracker. For a real recording $x$ and a trust level $\beta>0$, we target the generalized-Bayes (Gibbs) posterior

$$\pi_\beta(\theta \mid x) \propto p(\theta)\, \exp\!\big(-\beta\, \ell(x, g(\theta))\big),$$

where $p(\theta)$ is a smoothness prior on articulator trajectories and $\ell$ is a multi-resolution mel distance. The trust level matters because the simulator is misspecified: it has no nasal coupling, turbulence or soft tissue. A large $\beta$ pretends the simulator is exact and collapses onto one fit. A small $\beta$ admits that many trajectories explain real audio about equally badly.

We train a conditional normalizing flow $q_\phi(\theta \mid x, \beta)$, conditioned on $\beta$ as in Amortized Simulation-Based Inference in Generalized Bayes via Neural Posterior Estimation. A normalizing flow is an invertible network that turns Gaussian noise into samples from a learned density. We minimize

$$\mathcal{L}(\phi) = \mathbb{E}_{x \sim \text{real},\, \beta}\, \mathrm{KL}\big(q_\phi(\cdot \mid x,\beta)\,\|\,\pi_\beta(\cdot \mid x)\big) \;-\; \lambda\, \mathbb{E}_{(\theta,x)\sim \text{sim}} \log q_\phi(\theta \mid x, \beta{=}\beta_0).$$

The first term is amortized variational inference on real, unlabeled speech. It is possible only because the simulator is differentiable, so the gradient of $\ell$ flows through $g$. The second term is standard neural posterior estimation on simulated pairs with weight $\lambda$. It covers the modes that the mode-seeking first term tends to drop, and it is exactly where acoustic equivalence is known and checkable.

What is new relative to the closest work is the combination: amortized, $\beta$-conditioned posteriors, trained on real recordings rather than only simulator output, through a physical speech simulator, with no articulatory labels at any point before evaluation.

Research Program

Each prior paper plays a specific role in this program, so we position against them one at a time.

VocalTrax supplies the simulator and motivates the work. It shows that label-free inversion through Pink Trombone is feasible on TIMIT, AudioMNIST and VIVAE. It also documents what we attack: per-utterance optimization, a single local point estimate, and a persistent quality gap.

Learning Joint Articulatory-Acoustic Representations with Normalizing Flows constrains the design. It couples Pink Trombone tract images and spectrograms through a bijective GLOW flow, trained only on 36,081 synthetic static vowels. Its formant errors (F1 18.57%, F2 24.21%) and its one-to-one assumption tell us not to force a bijection between sound and tract. We instead model a conditional density over $\theta$ given $x$, which can be multimodal.

Automatic Posterior Transformation for Likelihood-Free Inference supplies the evidence that conditional autoregressive flows recover crescent-shaped, multimodal posteriors such as Two-Moons. BayesFlow supplies the summary-network-plus-conditional-flow architecture and simulation-based calibration checks. Both were validated only where the simulator is the truth, which is the assumption we drop.

Sun et al. supply $\beta$-conditioning. They train with forward KL on simulator draws, and they report two problems: importance-weight collapse far from $\beta=1$ (their Route B) and brittle Langevin tuning (their Route A). Our reverse-KL term uses the simulator's gradients on real inputs and needs neither reweighting nor synthesized off-manifold data. Whether this actually avoids their failure modes is an empirical question we test directly.

The build has three parts. The first is a summary network over mel frames and frozen WavLM-style self-supervised features. The second is a temporal conditional flow over parameter trajectories at 50 Hz, with coupling layers conditioned on a sliding context of summary vectors. The third is the JAX simulator in the loss. We will train on LibriSpeech and LibriTTS-R audio with no transcripts, and on simulated pairs drawn from a trajectory prior matched to VocalTrax's parameter ranges.

For evaluation only, we map simulator parameters to EMA coordinates with an affine readout. This is justified by Self-Supervised Models of Speech Infer Universal Articulatory Kinematics, which shows that affine maps align articulatory spaces across speakers and languages (within-group correlations above 0.85). The readout is fit on a controlled budget of EMA minutes. That budget is the axis on which the labeled-data claim lives.

There is a real risk here. Pink Trombone's tongue index/diameter and constriction parameters may not be affinely related to the six EMA coils. If the affine readout saturates well below EMA-supervised systems even with all EMA data, the physical parameterization is the bottleneck, and we will report that rather than hide it.

Baselines

The naive reference point is an affine readout from mel-cepstral features to EMA, fit with the same EMA budget. Self-Supervised Models of Speech Infer Universal Articulatory Kinematics reports roughly 0.66 correlation for acoustic features with full EMA supervision. Any proposal that fails to clearly exceed this at small budgets has no case.

The strongest label-efficient competitor is a supervised linear probe on a middle layer of a frozen self-supervised speech model. This is the approach of Cho et al. and of Articulatory Encodec (SPARC), which fits a linear head from WavLM layer 9 to 12 EMA coordinates and decodes with HiFi-GAN. With full EMA supervision the probe exceeds 0.8 correlation (Cho et al.). It is the strongest competitor because it already shows that self-supervised features encode articulation almost linearly. We run it with the same backbone, the same EMA minutes, and the same speaker splits, sweeping ridge strength with an equal tuning budget.

The strongest fully supervised reference is Speaker-Independent Acoustic-to-Articulatory Speech Inversion. It reports a Pearson correlation of 0.784 on combined MOCHA-TIMIT and HPRC EMA, using HuBERT features, adversarial trajectory losses and phoneme multi-task training. We quote this number rather than rerun it.

Beating these means one of two things. At equal EMA budget below roughly 30 minutes, our method should give higher unseen-speaker correlation than the SSL probe. Alternatively, it should match 0.784 with an order of magnitude less EMA than the supervised systems consumed.

For label-free inversion and resynthesis, the baseline is VocalTrax run per utterance with its published settings. It reports 48.7% automated digit-identification accuracy on AudioMNIST resynthesis, against 11.0% for the earlier gradient baseline. We match its simulator, its pitch tracker and its mel loss. No reported per-utterance VocalTrax runtime was found on Lacuna, so we measure wall-clock time ourselves on identical hardware. Beating it means comparable AudioMNIST accuracy and TIMIT mel distance at a small fraction of its per-clip cost, plus calibrated uncertainty that it cannot provide.

Four ablations isolate the key ingredients. The first is simulation-only neural posterior estimation, which isolates the value of training on real audio. The second is reverse-KL-only training, which isolates the simulated-pair term's role in keeping modes. The third fixes $\beta$, which isolates tempering. The fourth replaces the flow with a deterministic encoder trained on the same reconstruction loss, which isolates the value of a distribution over a point.

Why not just…?

Why not just train a deterministic encoder through the simulator on unlabeled speech, i.e., amortize VocalTrax directly? This is the most serious threat. It gets the speed, removes labels, and trains on real audio. It is literally our fourth ablation. If it matches our EMA correlation and resynthesis quality, the only remaining contribution is uncertainty, and we will say so. We therefore test whether uncertainty earns its keep in three ways. Do the true EMA positions, pushed through the affine readout, fall inside our 90% credible regions near 90% of the time? Does discarding high-variance frames raise correlation on the rest? And on simulated held-out data with known equivalent tracts, does the posterior place mass on each equivalent configuration while the deterministic encoder averages them into a shape that belongs to neither?

Why not just fit a small SPARC-style probe on whatever EMA exists? Because that is the regime we claim to improve, the comparison is the label-efficiency curve itself. If the self-supervised probe with five minutes of EMA already matches our method, then the physics and the posterior add nothing for accuracy. Their remaining value would lie in covering the glottal source and giving uncertainty, which EMA omits, as Cho et al. acknowledge. We would report the result as a negative finding on the labeled-data claim.

Why not just train neural posterior estimation on simulator output and apply it to real speech? Standard posterior estimation assumes simulated and real audio match. Pink Trombone audio is plainly unlike human recordings, and a summary network trained only on it will receive out-of-distribution inputs. The first ablation measures this directly as the drop in EMA correlation and in coverage from simulated test data to MOCHA-TIMIT.

Why not just use per-utterance optimization and accept the cost? For single clips this is fine. It yields no uncertainty, and it converges to local optima under acoustic equivalence. Its cost scales linearly with corpus size, which rules out processing LibriSpeech-scale data for downstream use. The amortization speedups reported elsewhere, 2,700 hours down to 3.7 seconds in BayesFlow and 12 hours to 0.1 seconds in gravitational-wave CVAE inference, show the scale of gain that is typical. We measure ours rather than assume it.

Proposed Experiments

The decisive experiment is the label-efficiency curve. We report unseen-speaker Pearson correlation on MOCHA-TIMIT and HPRC as the affine-readout EMA budget grows from one minute to the full training set. It compares our method, the SSL linear probe, the mel baseline, and all four ablations, with speaker-disjoint splits. The multilingual 62-speaker EMA corpus of Cho et al. tests cross-language transfer if it is accessible.

The second experiment is calibration. We run simulation-based calibration on held-out simulated trajectories across $\beta$, then compute empirical coverage of EMA positions on real speech. We expect some intermediate $\beta$ to give the best real-data coverage, and we report which one.

The third experiment is resynthesis. We compare against VocalTrax on AudioMNIST digit-identification accuracy (its reported 48.7%), on TIMIT mel distance, and on VIVAE emotion classification. We also report the word error rate of resynthesized LibriTTS-R. SPARC's 5.43% against 4.22% for ground truth is an upper reference that uses a neural vocoder, and we do not expect Pink Trombone audio to reach it.

The fourth experiment is an equivalence probe. On simulated clips engineered to share spectra, we count recovered modes. On real vowels with low second-formant frequency, we check whether posterior samples trade lip rounding against tongue backing.

A finding that the deterministic amortized encoder matches the flow on every metric except coverage would narrow the contribution to calibrated uncertainty. A finding that the SSL probe dominates at every EMA budget would falsify the labeled-data claim.