Touch as a Calibration Signal: Adapting Speech Inversion to New Speakers from Tongue–Palate Contact

Acoustic-to-articulatory inversion, which means estimating hidden tongue, lip and jaw movements from the sound of speech alone, works well on the speakers it was trained on and worse on everyone else. Vocal tracts differ in length, palate shape and dental arch, and the same sound can come from several different tongue postures. Electropalatography (EPG) measures something much narrower. The speaker wears a thin custom plate on the roof of the mouth, and an array of electrodes on it reports, frame by frame, where the tongue is touching. EPG is exact at the moments of contact and says nothing at all during open vowels, when the tongue drops away. We propose to use those brief, exact contact events as supervision at test time. A short session with a palate on a new speaker would correct a pretrained inversion model for that speaker. The claim we want to test is that the correction carries over to the frames EPG cannot see, including open vowels and later audio-only speech.

Robotic manipulation already has this structure. When a gripper closes on an object, the fingers block the camera. A tactile array then gives direct but spatially sparse information that vision cannot supply (Touch in the Wild reports 85% versus 25% success on test-tube racking with and without touch). The open problem in manipulation is the stretch just before contact, when vision is already blocked and touch has not registered yet. Speech has the same pattern: acoustics are least informative about tongue position at constrictions, which is exactly where the palate is touched, and touch is silent everywhere else. We treat contact as a sparse, exact anchor, and the no-contact stretches as the place where the proposal succeeds or fails.

Core Claim

A pretrained speaker-independent inversion model can be adapted to an unseen speaker using contact events alone. The adaptation step updates two things jointly. The first is a small set of model parameters. The second is a small, regularized description of that speaker's palate. On held-out utterances, the adapted model should improve articulator trajectory accuracy on frames with no contact, not only on frames with contact, and it should do so beyond what speaker embeddings, vocal tract length normalization or unsupervised test-time adaptation achieve. If the gains show up only at contact frames, the idea reduces to post-hoc clamping of the model's output and is not worth pursuing. The experiments are built so that outcome would be visible.

How This Relates to Prior Work

Two strands supply the pretrained inverters. Speaker-Independent Acoustic-to-Articulatory Inversion through Multi-Channel Attention Discriminator feeds WavLM-Large features, which come from a large self-supervised speech encoder, into a Conformer backbone trained with an adversarial discriminator. It reaches a Pearson correlation of 0.847 on HPRC, an electromagnetic articulography (EMA) corpus that tracks sensors glued to the tongue and lips. Speaker-Independent Acoustic-to-Articulatory Speech Inversion uses HuBERT features, speaker embeddings and an auxiliary phoneme-prediction task, reaching 0.784 on MOCHA-TIMIT. Both handle speaker variation at training time and never see the test speaker. We reuse them as frozen-then-adapted backbones.

The file cites two static speaker-handling alternatives that we treat as baselines. Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion warps the acoustic features to compensate for vocal tract length. Speaker Conditioned Acoustic-to-Articulatory Inversion Using x-Vectors conditions the inverter on a speaker embedding. Both correct for the speaker from audio alone and never observe the tongue.

The EPG side motivates the proposal. EPG2S fuses EPG with audio to generate and enhance speech. Its setting is a single male speaker with no articulatory targets and no treatment of the frames where contact is missing. Combined optical distance sensing and electropalatography documents that EPG reads zero during /e/, /a/ and /o/. It fixes this in hardware with optical sensors, one of which failed outright. Our proposal is a software response to the same blindness.

The test-time adaptation literature constrains the method design.

  • Tent minimizes prediction entropy by updating only the scale and shift parameters of normalization layers. That objective has no natural meaning for continuous regression outputs.
  • MemBN shows that per-batch normalization statistics collapse at batch size one: 90% error on CIFAR10-C, against 12.88% with a memory queue of past statistics. That result matters for streaming speech.
  • Examining Test-Time Adaptation for Personalized Child Speech Recognition reports that single-utterance adaptation lowers word error rate on child speech from 24.3% to 22.8%. The same paper finds that coughs and hesitations can corrupt unsupervised objectives.

Together these argue for an adaptation signal that is measured rather than self-generated. Contact is such a signal, and it is unaffected by acoustic noise.

The manipulation work supplies the method's template. Localizing the object contact through matching tactile features with visual map registers low-resolution pressure images against a prior map of the object with Bayesian filtering, localizing contact to within 5 mm after 4 to 6 touches. Our speaker-specific palate description plays the role of that map. The difference is that the map here is unknown and must be estimated jointly with the tongue.

Research Program

The central component is a differentiable contact head, a small network that takes the inverter's predicted articulator positions and the speaker's palate parameters and outputs a contact probability for each palate region. The palate parameters are a height offset and a curvature term per region, and they default to a population mean. The contact head is trained on source speakers and frozen. At test time, the new speaker reads for a fixed budget of seconds while wearing a palate. We then update two sets of parameters:

  • the normalization-layer scales and shifts of the inverter, plus an optional low-rank adapter;
  • the speaker's palate parameters $\pi$.

The objective is

$$ \mathcal{L}(\theta,\pi) = \sum_{t}\mathrm{BCE}\big(c_t,\; g(f_\theta(x_t),\pi)\big) + \lambda\,\lVert \pi-\bar\pi\rVert^2 + \mu\,\lVert f_\theta(x_t)-f_{\theta_0}(x_t)\rVert^2 . $$

The symbols are defined as follows:

  • $x_t$ is the acoustic input at frame $t$.
  • $c_t$ is the observed binary contact pattern.
  • $f_\theta$ is the inverter with adaptable parameters $\theta$, and $\theta_0$ are the pretrained values.
  • $g$ is the frozen contact head, and BCE is binary cross-entropy.
  • $\bar\pi$ is the population-mean palate.

The last term keeps the adapted predictions close to the pretrained ones everywhere, including no-contact frames. It prevents the model from satisfying the contact loss by distorting open-vowel postures.

The obvious weakness is identifiability. A predicted contact error can be absorbed either by lowering the palate or by raising the tongue. The palate prior carries that ambiguity. We will measure directly how often adaptation spends its correction on the palate parameters rather than on the tongue, by comparing recovered palate parameters against the true palate on synthetic speakers. Normalization statistics are accumulated in a memory queue following MemBN, so the method can run one utterance at a time.

EMA and EPG are rarely recorded together, so development proceeds in two stages. In the first stage, we derive a synthetic contact channel from EMA on HPRC and MOCHA-TIMIT. A region counts as in contact when the tongue-tip or tongue-body constriction degree, or the lip aperture computed from the sensor positions, falls below a threshold. We perturb this channel with speaker-specific palate offsets that the model does not see. This gives full kinematic ground truth everywhere, so no-contact frames can be scored. The cost is a contact channel far coarser than a real 62-electrode palate, which we treat as a lower bound on the information available. We will also check whether any release of MOCHA-TIMIT includes a synchronized palatography channel; the file does not establish this. In the second stage, we use real EPG, the paired audio and 64-electrode recordings of the TMHINT corpus used by EPG2S, together with a small new cohort recorded with the same palate hardware. On real EPG there is no kinematic ground truth, so the second stage tests transfer to audio-only use rather than trajectory accuracy.

Baselines

The do-nothing reference is the pretrained speaker-independent inverter of Speaker-Independent Acoustic-to-Articulatory Inversion through Multi-Channel Attention Discriminator, applied unchanged to the held-out speaker. It is the strongest reported system on our main benchmark, with PCC 0.847 and RMSE 2.467 on HPRC. The previous speaker-independent best it reports is PCC 0.810. On MOCHA-TIMIT the reference number is PCC 0.784 from Speaker-Independent Acoustic-to-Articulatory Speech Inversion. We will retrain it ourselves under our leave-one-speaker-out split so that all methods share identical folds. We will report our reproduction next to the published figure.

The static-normalization baseline adds both x-vector conditioning (Speaker Conditioned Acoustic-to-Articulatory Inversion Using x-Vectors) and vocal tract length normalization (Vocal Tract Length Normalization for Speaker Independent Acoustic-to-Articulatory Speech Inversion) to the same backbone. For fairness, the speaker embedding and warping factor are estimated from exactly the adaptation audio our method receives. It is the strongest competitor because it uses the same adaptation audio without any hardware.

The unsupervised test-time adaptation baseline applies SUTA-style entropy minimization to the backbone's auxiliary phoneme head, keeps normalization statistics in a MemBN queue, and updates the same parameter subset as our method. It receives the same audio, the same number of gradient steps and the same learning-rate sweep. It answers whether measured contact adds anything over self-generated targets.

The post-hoc clamping baseline leaves the model untouched and projects its output onto the contact constraints only at frames where contact is observed. This is the naive way to use EPG.

Two ablations isolate the key ingredients. One removes the palate parameters, forcing all correction into the network. The other removes the anchoring term $\mu$.

Beating the baselines means two things. First, a PCC improvement on no-contact frames of held-out utterances over the best of the non-contact baselines, stable across speakers in a paired test. Second, matching clamping on contact frames while exceeding it on no-contact frames, where clamping by construction does nothing. No reported result was found on Lacuna for inversion with any test-time adaptation, so the size of the effect is open.

Why not just…?

Why not just condition on x-vectors or normalize vocal tract length, and skip the palate? This is the real threat, because it needs no hardware. The case for contact is that these transforms estimate the speaker from audio. Audio is where the one-to-many ambiguity between tongue and lip posture lives, so a better speaker estimate from audio need not resolve it. The file shows that frozen speech features still leave large speaker gaps. Phone error rate rises from 0.288 to 0.336 for non-native speakers in 75-Speaker Annot-16, and word error rate is 30.5% for children with off-the-shelf features. That is suggestive, not decisive. The matched-budget comparison settles it. If x-vectors plus length normalization match our no-contact gains, the palate is not earning its cost and we will say so.

Why not just run Tent- or SUTA-style unsupervised adaptation? Entropy has no clean meaning for trajectory regression. Proxying it through a phoneme head adapts toward phonetic confidence, not geometric accuracy. The child-speech study reports that non-speech sounds corrupt that objective. Contact labels are measured and unaffected by acoustic noise. The comparison uses an identical parameter subset and identical memory-queued statistics, so any difference is attributable to the signal itself.

Why not just overwrite the inverter's output wherever the palate reports contact? Clamping captures everything contact can offer at contact frames. If that were all contact offered, adaptation would be pointless. The decisive number is therefore the no-contact-frame gain on utterances never seen during adaptation. Clamping scores zero there by construction.

Why not just fine-tune the whole inverter on predicted contact? With minutes of binary labels, full fine-tuning can fit contact by distorting open-vowel postures that contact never checks. We include full fine-tuning at the same step budget. We expect it to improve contact-frame fit while degrading no-contact frames. If it does not, the anchoring term is unnecessary and we will drop it.

Proposed Experiments

The main experiment is leave-one-speaker-out on HPRC's eight speakers, with synthetic contact and randomized palate offsets. We sweep adaptation budgets of 10, 30, 60 and 120 seconds and report PCC and RMSE separately for contact frames, no-contact frames and open vowels, all on held-out utterances. A harder cross-corpus condition trains on MOCHA-TIMIT and adapts to HPRC speakers, where the speaker shift is larger.

A synthetic check tests identifiability. We compare recovered palate parameters against the injected offsets, and we report how the correction splits between palate and tongue as the prior strength $\lambda$ varies.

A streaming condition runs adaptation one utterance at a time, with and without the memory queue. It tests whether the MemBN failure at batch size one reappears in regression.

The real-hardware check uses TMHINT and the new cohort. We adapt on one set of sentences with the palate, then predict contact from audio alone on new sentences, which is the use case of a learner who later practices without the palate. Metrics are frame-level contact F1 and error in the timing of closure onset, compared against the unadapted and x-vector baselines. EPG2S reports no numbers on TMHINT, so no reported result was found on Lacuna for audio-to-contact prediction there, and we will establish the baseline ourselves.

The project is falsified if no-contact gains fail to exceed the x-vector baseline in the main experiment at any budget. The same holds if gains vanish when the synthetic contact channel is made as coarse as three regions, since real hardware would then not rescue it.