Pinning the Articulatory Frame: Phonetic Anchors and Tension Tests for Pooling Speech Inversion Corpora

Acoustic-to-articulatory inversion means recovering how the tongue, lips and jaw moved from a recording of the voice alone. The best systems are trained by pooling articulatory corpora recorded with different machines, sensor layouts and speakers. Pooling helps on new data but quietly hurts on familiar data. That trade-off reflects a confusion the field has never separated cleanly. A tongue sensor sitting 3 mm higher in one speaker could mean a different palate, a different sensor placement, or a different gesture, and the regression target cannot tell these apart. Particle physicists face the same structure when one measurement leaves a "flat direction" of equally good parameter values. They break it by combining measurements whose flat directions cross, and they test whether the combination is legitimate before trusting it.

We propose to do both for speech inversion, using the tool experimental phonetics already relies on: deliberately limited phonetic scope. A small set of phonetically pinned events serves as anchors. These are lip contact during /p b m/, tongue-tip closure for /t d/, dorsal closure for /k g/, the high-front tongue target of /i/, and maximal opening for /a æ/. They fix each speaker's articulatory coordinate frame independently of what that speaker happened to say. A parameter goodness-of-fit test across anchor classes then tells us, speaker by speaker, whether a single shared articulatory space is consistent with the data or whether pooling that speaker will do harm.

Core Claim

The cross-corpus and cross-speaker degradation in pooled inversion is partly a degeneracy between per-speaker frame parameters (offset, scale and rotation of sensor coordinates or tract variables) and articulatory content. Content-independent anchors from limited-scope phonetic events break this degeneracy better than whole-corpus statistics. Tension between anchor classes is a measurable signature of when a speaker's frame cannot be reconciled with the shared space. The deliverables are a calibrated per-speaker normalization and a diagnostic that predicts which speakers or corpora will cause negative transfer when added to training. The diagnostic is the main contribution; neither existing pooling method nor physical global fitting provides it.

Positioning

Three lines of inversion work motivate the gap. Multi-Corpus Acoustic-to-Articulatory Speech Inversion maps XRMB, EMA-IEEE and MOCHA-TIMIT onto nine tract variables and trains a network with one output head per corpus. In that study an XRMB-only model collapses to 0.453 correlation on EMA-IEEE, and pooling lowers native EMA-IEEE correlation from 0.826 to 0.819. Per-corpus heads avoid ever committing to a shared frame. We instead estimate the frame explicitly and test whether sharing it is warranted.

Speaker-Independent Acoustic-to-Articulatory Speech Inversion conditions a HuBERT-based multi-task model on learned speaker embeddings. It reaches 0.784 Pearson correlation on combined MOCHA-TIMIT and HPRC, but its resynthesis distortion remains high. Its speaker embedding absorbs frame differences implicitly, with no way to tell an anatomical offset from a habit.

Training Articulatory Inversion Models for Interspeaker Consistency introduces a minimal-pair separability benchmark whose scores track downstream synthesis word error rate (r = −0.96 English, −0.89 Russian). It also shows that a consistency loss helps English and hurts Russian. That result is exactly the kind of negative transfer a tension test should anticipate.

Two papers constrain the design. Using phonetic constraints in acoustic-to-articulatory inversion scores candidate shapes from Maeda's model against ideal phonetic domains. It is still left with a 54 Hz average F1 mismatch that the authors attribute to the 2D-to-3D area conversion. We take this as a warning against making a physical acoustic simulator the center of the fit. Our anchors live in measured kinematic space, not in simulated acoustics. Some articulatory and acoustic changes associated with emphasis in spoken English shows that speakers split into jaw-dominant and pitch-dominant strategies under emphasis. Open-vowel anchors are therefore taken only from unemphasized tokens, with emphasis status as a covariate where it is available. Supportively, Lexical stress affects lenition: The case of Italian palato-alveolar affricates finds acoustic lenition differences between speaker groups but no systematic tongue-tip target differences. Constriction targets are thus plausibly stable enough to serve as anchors.

The statistical method is borrowed directly. The parameter goodness-of-fit used in Gallium Anomaly: critical view from the global picture of νe and ν̄e disappearance showed that the Gallium deficit and reactor data cannot share a sterile-neutrino solution (GoF_PG below 0.1%). The likelihood-ratio tension test in Quantifying the tension between the Higgs mass and (g−2)μ in the constrained MSSM plays the same role. We apply this logic with anchor classes in place of experiments and speaker frames in place of physics parameters.

Research Program

We first extract anchors. Each corpus is phone-aligned with forced alignment from its transcripts. For each speaker we take one articulator measurement per anchor event:

  • Lip aperture at its minimum within bilabial closures.
  • Tongue-tip height at its maximum within /t d/ closures.
  • Tongue-dorsum height at its maximum within /k g/ closures.
  • Dorsum position at the /i/ midpoint.
  • Jaw and lip opening at the midpoint of /a æ/.

Each speaker $s$ has frame parameters $\phi_s$: an affine map (offset, per-variable scale, small rotation in sensor space) from that speaker's coordinates to a shared canonical space. Each anchor class $k$ has a canonical target $\mu_k$ with between-speaker spread $\Sigma_k$. The anchor likelihood for a class is the product over that speaker's tokens of a Gaussian in canonical space, giving a per-class chi-square $\chi^2_{s,k}(\phi_s)$. Targets and spreads are estimated hierarchically across all speakers, and $\phi_s$ is obtained by profile likelihood.

The tension test asks whether one frame serves all anchor classes for a speaker. Let $K$ be the number of anchor classes. Let $\chi^2_{s,\min}$ be the minimum of $\sum_k \chi^2_{s,k}(\phi_s)$ with a single $\phi_s$, and $\chi^2_{s,k,\min}$ the minimum for class $k$ fitted alone. We compute

$$\chi^2_{\mathrm{PG},s} = \chi^2_{s,\min} - \sum_{k=1}^{K} \chi^2_{s,k,\min}.$$

This statistic is referred to a chi-square with degrees of freedom equal to the summed per-class fitted parameters minus the parameters of the joint fit. Fitting each class separately is often underdetermined; where it is, we restrict each class to the frame components it actually constrains, as in the neutrino analyses. Because every class constrains a different subset of frame components (lips versus tongue tip versus dorsum), the class-wise flat directions cross, which is the degeneracy-breaking argument carried over.

A low GoF_PG flags a speaker whose frame is not affine-consistent across articulators. Plausible causes are a misplaced sensor, a nonlinear palate difference, or a dialect-specific target. Because the test isolates the anchor class responsible, it gives the localized failure analysis that aggregate RMSE hides.

We then canonicalize all training targets with the fitted frames. We train one fixed inversion architecture, the HuBERT-feature multi-task model of the speaker-independent paper, with a single shared output space rather than per-corpus heads. A second variant excludes or down-weights flagged speakers according to their GoF_PG.

Baselines

The naive reference point is the same architecture trained on pooled tract variables with no per-speaker normalization. It shows how much of the pooling problem is frame mismatch at all.

The strongest conventional normalization is per-speaker z-scoring of every articulatory channel over all of that speaker's frames. This is an affine frame fit using whole-corpus moments instead of anchors, and it is the most direct competitor to our first deliverable.

The strongest existing methods are two published designs, each reimplemented on our exact speaker splits and features:

Our splits and preprocessing will differ from those papers, so the published figures serve as sanity checks. All comparisons use our own runs, with identical training data, network size, number of updates and hyperparameter search budget, and each method tuned on the same validation speakers.

The key ablation replaces anchor-fitted frames with frames fitted by the same affine model to all aligned frames. The model is the same; only the anchors are removed. This isolates the claim that limited-scope events, not the affine form itself, carry the benefit. A second ablation removes one anchor class at a time to show which classes break which flat directions.

Beating the strongest baseline means two things on held-out speakers. First, higher leave-one-corpus-out correlation and minimal-pair separability without any native-corpus loss relative to single-corpus training. That is, closing the 0.826 versus 0.819 gap in our runs rather than trading it for transfer.

Why not just…?

Why not just z-score each speaker's channels? This is the real threat, because it is an affine normalization that is nearly free. Its weakness is that a speaker's channel means depend on what was said and how fast. XRMB, IEEE sentences and MOCHA-TIMIT have different phonetic content, and HPRC mixes speaking rates. A speaker with fewer open vowels gets a shifted jaw mean for reasons that have nothing to do with anatomy.

We test this directly. We take a speaker, artificially drop subsets of sentences to skew vowel and consonant composition, and measure how far each normalization's frame estimate moves; the anchor-based estimate should barely move. If z-scoring matches anchors on both this test and downstream metrics, the normalization claim fails. The diagnostic contribution would then stand alone, and we would report it that way.

Why not let a learned speaker embedding absorb the frame? This is the approach of Speaker-Independent Acoustic-to-Articulatory Speech Inversion, and it may match our correlations. An embedding cannot say whether a speaker is consistent with the shared space, and it can absorb genuine articulatory habits along with sensor offsets. The negative English-to-Russian transfer in Training Articulatory Inversion Models for Interspeaker Consistency shows that implicit pooling can fail silently. We compare correlation head to head. We also compare whether GoF_PG, which the embedding model lacks any analogue of, predicts per-speaker negative transfer.

Why not do a full Bayesian fit of a physical vocal tract model, as the global-fitting analogy suggests? Potard and Laprie's 54 Hz residual F1 error shows that the acoustic simulator itself would dominate the likelihood. Tension would then point at the simulator rather than at speakers. Restricting the fit to measured kinematics at limited-scope events avoids this. We give up acoustic-articulatory coupling, which remains future work.

Proposed Experiments

We use XRMB, EMA-IEEE, MOCHA-TIMIT, HPRC and the native-English portion of EMA-MAE, with whole speakers held out per fold.

Pooling experiment. We train every baseline and our model under leave-one-corpus-out and within-corpus speaker-held-out protocols. Metrics are Pearson correlation and RMSE on tract variables, plus the minimal-pair separability benchmark of McGhee et al.

Diagnostic validation, synthetic. We inject known non-affine distortions into a subset of clean speakers, such as a displaced tongue-tip sensor or a warped dorsum axis. We measure whether GoF_PG flags them and attributes the problem to the right anchor class.

Diagnostic validation, real. For each real speaker we measure the change in held-out performance on the other speakers when that speaker is added to training. The decisive check is whether low GoF_PG predicts negative transfer better than chance and better than simple heuristics, namely per-speaker training loss and z-score frame instability.

Outcomes. If anchored frames improve pooling and GoF_PG predicts harm, the paper shows that pooling legitimacy in speech inversion can be tested before training. If only the diagnostic works, that is still a usable result. If neither holds, then frame mismatch is not what limits pooled inversion, which narrows the search for what does.