Calibrating Articulatory Inversion to a New Speaker from a Few Phonetically Chosen Frames

Articulatory inversion means recovering the movements of the tongue, lips and jaw from the audio of speech alone. It works well when a model is trained on the same speaker it is tested on, and much worse on a new speaker. We propose to close most of that gap with a small, closed-form correction. The new speaker contributes a few dozen frames of sensor-tracked articulation. The frames are chosen by a simple phonetic rule: the middles of corner vowels (/i/, /a/, /u/) and the moments of lip closure in /p, b, m/, where the true articulator position is nearly fixed across speakers and contexts. A weighted least-squares fit on those frames, with no gradient descent and no retraining, corrects the output of a frozen speaker-independent network.

The rationale is that part of the cross-speaker error cannot be heard. Where a sensor was glued on the tongue, and how each speaker's head-centered coordinate frame is oriented, shift the recorded coordinates without changing the sound. No audio-only method, however strong, can recover that part. A low-dimensional geometric correction can. The experimental phonetics tradition of restricting stimuli to a few well-understood sounds tells us which frames make that correction well-determined. In this proposal, least squares supplies the estimator and limited phonetic scope supplies the data-selection rule that makes the estimate trustworthy from very little data.

Core Claim

Our claim is that a per-speaker affine map (a linear transform plus offset for each sensor's two coordinates) can be fitted by weighted least squares on fewer than 100 phonetically anchored frames, and that it recovers a substantial share of the gap between speaker-independent and speaker-dependent inversion. We further claim that the anchored frames matter beyond their number: the same number of randomly chosen frames, or the same duration of arbitrary speech, should give a worse-conditioned and less accurate fit.

The weights come from phonetic knowledge rather than tuning. For each phone and each articulator channel, the weight is the inverse of that channel's within-phone variance across training speakers. This makes "critical" articulators dominate the fit, such as the lips during /m/ or the tongue dorsum during /i/. Statistical identification of critical, dependent and redundant articulators showed that roughly four articulators are critical for each phone and that models built around this distinction need 40–43% fewer parameters. We reuse that finding as the weighting scheme.

The estimator is

$$\min_{A_s, b_s} \sum_{t \in \mathcal{T}} \left(y_t - A_s \hat{y}_t - b_s\right)^\top W_{p(t)} \left(y_t - A_s \hat{y}_t - b_s\right),$$

where the symbols are as follows. $\hat{y}_t$ is the frozen network's predicted sensor coordinates at frame $t$, and $y_t$ is the measured coordinates. $A_s$ is the per-speaker matrix, block-diagonal with one 2×2 block per sensor, and $b_s$ is the per-speaker offset. $\mathcal{T}$ is the anchor frame set, and $W_{p(t)}$ is the diagonal weight matrix for the phone at frame $t$.

The solution is closed-form. Its parameter covariance, $(X^\top W X)^{-1}$ scaled by the residual variance, is available for free. We use that covariance in two ways: as a stopping rule ("enough anchors collected") and as a diagnostic of which sensors are poorly determined.

Relation to Prior Work

Two strands motivate the gap.

The first is speaker normalization. Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion warps the acoustic frequency axis, choosing the warp by maximum likelihood over unconstrained speech. That gains 7% absolute correlation, yet the system stays about 16% poorer than speaker-dependent systems. The same paper reports that adapting on target-speaker data closes the gap entirely, which tells us target-speaker articulatory data is decisive. We ask how little of it is needed, and which frames.

The second strand is the strong neural model. Speaker-Independent Acoustic-to-Articulatory Speech Inversion uses self-supervised HuBERT features and speaker embeddings to reach 0.784 Pearson correlation on 16 speakers. Its corrections operate entirely in the acoustic domain, so they cannot see sensor placement. We use a model of this kind as the frozen backbone rather than competing with it.

Three papers supply method and constraints.

Acoustic-to-articulatory inversion based on local regression fits local least-squares maps over about 1000 acoustically nearest neighbors drawn without phonetic conditioning. It reaches 1.52 mm and 0.78 correlation on one speaker, but is 100 times slower than global mixture regression. We take its lesson that least squares is competitive when the fitting set is coherent. We differ by fitting one global per-speaker map on phonetically selected frames instead of one map per test frame on acoustically selected frames.

Estimating the vocal-tract area function from formants using a sensitivity function and least square and DC-constrained linear prediction for glottal inverse filtering are the closest prior pairings of least squares with restricted phonetic scope. They use four static vowels and sustained /a/ respectively, and in both the restriction serves only to validate a single-speaker signal model. We instead make phonetic restriction the selection rule for a cross-speaker calibration set, and we test it on running speech.

Vocal tract length estimation based on vowels is a constraint on our design. It also uses vowels plus a linear calibration, reaching R² of 0.82 against MRI with about 6 mm standard error. That residual shows vowel-only acoustic estimates of anatomy stay coarse. This is why we calibrate against measured articulation rather than inferring anatomy from audio.

Research Program

Backbone. We train a speaker-independent inversion model in the style of the HuBERT-based system above. Training uses leave-one-speaker-out splits over the combined MOCHA-TIMIT and Haskins Production Rate Comparison (HPRC) EMA data (electromagnetic articulography, which tracks small sensors glued to the articulators).

Anchor selection. We forced-align each held-out speaker's sentences. We then select anchor frames from the existing read material: corner-vowel midpoints, bilabial closure midpoints, and alveolar closures for the tongue tip. This means no new recording protocol is needed. "Limited phonetic scope" is applied as a rule for choosing frames from ordinary speech, not as an artificial stimulus set.

Calibration. We fit the weighted affine map on budgets of 10, 25, 50, 100 and 400 anchor frames.

Evaluation. We evaluate on the speaker's remaining sentences.

Analysis. We measure what fraction of the speaker-independent gap is affine at all. To do this, we fit the same map on all of the held-out speaker's test data, as an oracle. This oracle bounds what any calibration of this form can achieve, and tells us early whether the idea is worth pursuing.

Baselines

Uncorrected backbone. The naive reference point is the frozen speaker-independent backbone with no correction, evaluated on the held-out speaker. Every calibration method is reported as the share of the gap it closes between this point and the speaker-dependent upper bound.

Speaker-dependent upper bound. We train the same architecture on the held-out speaker's own training sentences. For context, the strongest reported speaker-dependent figure in the file is 1.08 mm RMSE and 0.82 correlation for Articulatory-WaveNet on EMA-MAE, from Bozorg & Johnson. That result is on a different corpus, so we compare against our own speaker-dependent run.

Strongest speaker-independent method. This is the HuBERT-based model from Speaker-Independent Acoustic-to-Articulatory Speech Inversion, which reports 0.784 Pearson correlation on the 16-speaker combined EMA set. The file lists no released code, so we reimplement it. We report our reproduction's number next to 0.784 so readers can judge the gap. Beating it means exceeding our reproduction's held-out-speaker correlation by a margin larger than the across-fold standard deviation, at a calibration budget of 100 frames or fewer.

VTLN acoustic adaptation. Following Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion, we run VTLN on the same held-out speaker, using as much audio as it wants.

Neural fine-tuning. We fine-tune the backbone's final layers on exactly the same frames our method uses. Early stopping is done on a frame split, and the hyperparameter search budget is matched.

Ablations. The ablations isolate the key ingredient. Random frames replace anchor frames, at equal count and at equal duration. Uniform weights replace critical-articulator weights. A per-channel offset only replaces the full affine map.

Why not just…?

Why not just fine-tune the network on the calibration data? This is the most serious threat. With 50–100 frames, end-to-end fine-tuning should overfit, but a closed-form refit of only the last linear layer is a respectable competitor. We run that last-layer refit on both anchored and random frames. If it matches our affine map, the contribution reduces to the anchor-selection rule, which the random-versus-anchored comparison then tests on its own.

Why not just use more ordinary speech instead of choosing phones? It is possible that frame count, not frame identity, drives the gain. The equal-duration random-frame ablation answers this directly. The least-squares covariance gives a mechanistic check: anchored sets should yield a smaller condition number of $X^\top W X$ per frame. If random frames do just as well, we will report that phonetic scope adds nothing here.

Why not just rely on acoustic speaker normalization or speaker embeddings? VTLN leaves a 16% correlation gap in Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion. More fundamentally, sensor placement offsets are acoustically invisible. The oracle-affine analysis quantifies how large that invisible component is. If it is small, this objection wins and the proposal narrows to a cheaper alternative to collecting full speaker-dependent data.

Proposed Experiments

We run leave-one-speaker-out over 16 speakers from the combined MOCHA-TIMIT and HPRC data. Metrics are RMSE in millimeters and Pearson correlation per channel, averaged over channels. We also report correlation separately on critical and non-critical frames, using the critical-articulator labels.

The decisive checks are three. First, the oracle affine fit must close a meaningful share of the gap. If it closes less than a third, we stop. Second, at 50 anchored frames, the weighted affine map should close most of the oracle's share. It should also beat both last-layer refit and random frames at equal count, with differences larger than fold-to-fold variation. Third, the predicted parameter covariance should track actual calibration error across budgets, which would validate it as a stopping rule.

As a secondary test of transfer, we apply the procedure to the 10 X-ray Microbeam speakers used in Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion. Those recordings track gold pellets rather than electromagnetic sensors, so they test whether anchored calibration transfers across tracking technologies.