Listening to Resolve Ambiguity in Articulatory Inversion

Recovering how a speaker's tongue, lips, jaw and larynx moved from a recording alone is called acoustic-to-articulatory inversion. It is ill-posed: several different mouth shapes can produce almost the same sound. Current inversion systems settle this ambiguity by themselves, and they tend to fail where it matters most. McGhee et al. show that models scoring well on trajectory correlation can still blur phonemic contrasts, and that the blurring predicts poor intelligibility after resynthesis (Training Articulatory Inversion Models for Interspeaker Consistency). The same models degrade on non-native speakers and on languages other than English. We propose to resolve inversion ambiguity with a few well-chosen listening questions. The system builds a small set of plausible vocal-tract shapes for an ambiguous sound and renders each through the Maeda articulatory model, a seven-parameter description of the vocal tract derived from X-ray films whose parameters (jaw, tongue body, tongue tip, lips, larynx) can only reach anatomically possible shapes. It then plays the renderings to a listener who speaks the language and asks which one sounds like the original. The answer is spread to every token of that sound, and to related sounds, for that speaker.

The idea borrows its machinery from interactive ontology matching. That field found that a handful of expert answers, placed at the points where automatic matchers disagree and propagated through the structure being aligned, can recover much of what automation misses (Actively Learning Ontology Matching via User Interaction; LogMap 2.0). Our structure is a speaker's phoneme inventory. The items being aligned are regions of Maeda parameter space. The human is a listener, not a phonetician tracing X-ray frames.

Core Claim

Two things make listener judgments the right signal. Human hearing is good at telling apart the phonemic categories that inversion models merge. Maeda's physical constraints ensure that every candidate the listener hears is a shape a real vocal tract could make. Given both, we claim that a budget of roughly 20 to 50 forced-choice listening queries per speaker, chosen where candidate inversions disagree about phoneme identity and propagated along shared phonetic features, will improve two things for non-native and non-English speakers: phonemic separability (minimal-pair classification) and geometric accuracy against real-time MRI. The improvement should exceed what the same budget achieves with random queries, with no propagation, or with an automatic phone recognizer answering in place of the human.

That last comparison decides whether the human matters at all. We state it up front: if a multilingual machine listener matches human listeners, the correct conclusion is that the interactive loop is useful but the human in it is not.

Relation to Prior Work

Three inversion lines motivate the gap. Articulatory-WaveNet models trajectories autoregressively and reaches low error for a single known speaker (Acoustic-to-Articulatory Inversion with Deep Autoregressive Articulatory-WaveNet). The speaker-independent HuBERT-based inverter adds phoneme prediction as a second training target (Speaker-Independent Acoustic-to-Articulatory Speech Inversion). SPARC maps any voice onto one template speaker's articulator traces through a linear probe on WavLM (Coding Speech Through Vocal Tract Kinematics). All three resolve non-uniqueness autonomously, with no mechanism to ask anyone when the solution is degenerate.

McGhee et al. constrain the design. Their negative result, that correlation does not track separability, rules out correlation as our success metric. Their minimal-pair measure becomes our primary evaluation. Their finding that consistency training helped English and hurt Russian identifies the regime in which a language-specific signal should help most.

The ontology-matching papers supply the method. From Shi et al. we take query selection at contention points and propagation down a hierarchy. From LogMap 2.0 we take consistency checking and repair during matching rather than afterwards. From Duan et al. we take the finding that user feedback can tune a matcher to one idiosyncratic pair rather than relying on a universal template (One Size Does Not Fit All). We also address a limitation that Shi et al. leave open: their method assumes users never err. Our propagation is soft and models listener noise.

The closest human-involved speech work confines the human to offline data preparation. Laprie et al. trace contours on cine X-ray before driving a Maeda-based simulation (Articulatory copy synthesis from cine x-ray films). The Annot-16 team corrects forced alignments and seeds MRI segmentation (75-Speaker Annot-16). In neither case does the human sit inside the inversion loop.

Research Program

The base inverter is a mixture-output network. Its input is frozen WavLM features, following SPARC. Its output is a small mixture of Maeda parameter vectors per frame, rather than one point, so that ambiguity is represented explicitly. It is trained on two sources. The first is audio rendered by the Maeda forward model: sagittal contour, then area function, then acoustic simulation, as in the Laprie copy-synthesis chain. The second is EMA corpora (MNGU0, MOCHA-TIMIT, HPRC), with articulator traces mapped to Maeda parameters by fitting.

We handle the voicing blind spot explicitly. The Annot-16 paper shows tract shape cannot separate /p/ from /b/. We therefore take voicing and pitch from the signal, using glottal-source decomposition (Complex cepstrum-based decomposition of speech for glottal source estimation) plus pitch and loudness tracks, and never ask listeners about it.

For each speaker, the unit of alignment is a phoneme-by-mode cell: the set of tokens of one phoneme, together with the distinct shape modes the inverter proposes for them. A cell is a contention point when resynthesis from its competing modes is identified as different phonemes by an automatic phone recognizer. The competing modes are rendered with the speaker's own pitch and loudness contour. We score a cell by expected leverage,

$$ L(c) = \sum_{t \in T(c)} \Pr\big[\text{phoneme decision at token } t \text{ flips} \mid \text{answer to } c\big], $$

where $T(c)$ is the set of tokens, across the speaker's data, whose posteriors would be updated by an answer about cell $c$. The set includes tokens of phonemes sharing a place or manner feature, which is how propagation enters the score.

A query plays the original token and two or three candidate renderings. The listener chooses the closest match or "none." The answer updates a per-speaker prior over the target region for that phoneme. It propagates with decaying weight along a phonetic feature hierarchy: an answer fixing tongue-tip place for /t/ also shifts /d/ and /n/. This is the analogue of propagating from "Academic Staff" to its sub-concepts.

Listener error is modeled as a per-listener flip rate. We estimate it from catch trials in which one rendering is the original resynthesized. A consistency pass in the spirit of LogMap then flags contradictory answers for re-query. An example is a front-place answer for /s/ alongside a back-place answer for /ʃ/ in the same speaker.

Baselines

The naive reference is the base mixture inverter with no queries, taking the most probable mode. This measures how much ambiguity costs before any intervention.

The strongest existing method for cross-speaker articulatory recovery is SPARC. It reports 5.43% WER on unseen LibriTTS-R speakers against 4.22% for real audio, and 3.73% on VCTK (Coding Speech Through Vocal Tract Kinematics). We run the released model on the same test utterances and map its EMA output to Maeda parameters with the same fitting used for our training data, so that both systems are scored in one parameter space.

For trajectory recovery we also run the speaker-independent HuBERT inverter, which reports Pearson correlation 0.784 on combined EMA data (Speaker-Independent Acoustic-to-Articulatory Speech Inversion). Articulatory-WaveNet's 1.08 mm RMSE on EMA-MAE (Articulatory-WaveNet) is speaker-dependent and serves only as a within-speaker ceiling. For minimal-pair accuracy on Russian, no reported result was found on Lacuna for the absolute values in McGhee et al. We therefore re-run their protocol on all systems.

The competitor that most threatens the claim is the machine-listener loop. It is identical to our system, including query selection, propagation and budget, except that a multilingual phone recognizer answers each query. A second machine baseline uses the same selection score but answers with SSL-feature distance to the original token.

Ablations isolate each ingredient. One uses random query selection at equal budget. One removes propagation, updating only the queried cell. One removes the noise model and treats answers as certain. One is an oracle that answers from MRI-derived Maeda fits, giving an upper bound on what selection can achieve.

Every variant receives the same pretrained inverter, the same training data, and the same number of queries per speaker, with hyperparameters tuned on held-out native English speakers only. Beating the baselines means a statistically significant gain in minimal-pair accuracy on Russian and non-native speakers over both the no-query system and the machine-listener loop, at a budget of 50 queries or fewer, without worsening native-speaker results.

Why not just…?

Why not just use a self-supervised model with a linear probe, as SPARC does? SPARC is strong on English, and we expect it to stay competitive on native speakers. However, the SPARC authors themselves note that its underlying representations carry language bias. McGhee et al. show that high-correlation probes can still merge contrasts, and that consistency training degraded Russian. Our no-query condition is essentially this approach with explicit modes, so the experiments measure directly what the queries add on top of it.

Why not just add a phoneme classification loss or a minimal-pair loss during training? The speaker-independent inverter already trains with phoneme prediction as a second target, and it still yields high resynthesis distortion and English-centric behavior. A training loss can only enforce contrasts present in the training languages. It cannot tell the model that a particular non-native speaker realizes /r/ differently. This is a real alternative, so we include a variant fine-tuned with a minimal-pair loss on the adaptation speaker's untranscribed audio, using pseudo-labels, as an extra baseline.

Why not just let an automatic recognizer answer the queries? This is the most serious threat, and it is why the machine-listener loop is a first-class baseline. Our hypothesis is that recognizers inherit the same English bias as the inverter's features, and so fail exactly where queries are needed: Russian palatalization contrasts and non-native realizations. If the machine loop matches humans, we report that the contribution is the interactive selection-and-propagation loop, not human judgment.

Why not just have phoneticians annotate more MRI offline? The Annot-16 effort produced only 160 hand-traced ground-truth frames across 16 speakers, which shows how costly tracing is. LogMap 2.0 cut expert review from 2,692 to 585 candidate mappings (a 78% reduction) by placing the human inside the loop. Listening queries also need no imaging and no expertise, so they apply to speakers who will never enter a scanner.

Proposed Experiments

The first experiment uses the Annot-16 non-native speakers, the group on which rt-MRI phone error rises to 0.336 from 0.288 for native speakers (75-Speaker Annot-16). We fit Maeda parameters to MRI contours, using the 160 hand-traced frames plus SPAM segmentations (about 84% IoU on unseen subjects) as reference. We then measure parameter error and constriction-location error after 0, 10, 20 and 50 queries, answered by native-English listeners. Following Shi et al., we expect most of the gain within the first ten queries. If the learning curve is flat across the first 20 queries, that would count against the claim.

The second experiment uses RUSLAN. We apply the McGhee minimal-pair protocol and resynthesis WER, with native-Russian listeners answering queries. The decisive check is whether human-answered loops beat the machine-listener loop by more on Russian than on English. A widening gap supports the claim that human perception supplies language-specific knowledge the models lack.

The third experiment is a noise stress test. We inject 10% to 30% flipped answers and confirm that the noise model and consistency repair keep propagation from cascading errors, a failure mode left unaddressed in the original active-matching work.

Across all experiments, the proposal succeeds if human-answered queries improve separability and MRI agreement beyond random, non-propagated and machine-answered queries at equal budget. It fails informatively if the oracle is far ahead of humans, which would indicate that listeners cannot resolve the ambiguities Maeda synthesis presents to them.