Inferring Vocal Tract Dynamics from Speech Signals

This direction is merged into another direction: acoustic-to-articulatory inversion. Its last version stays readable and citable here.

Version 1 of 146 papers61 conceptsfrozen 2026-10-04

When we listen to a person speak, we are hearing the acoustic byproduct of a complex physical dance. The tongue, lips, jaw, and velum move in coordinated trajectories to shape the air expelled from the lungs. While traditional speech processing focuses on the resulting sound waves, a growing body of research suggests that the most robust information lies in the "source"—the physical kinematics of the vocal tract.

By "hallucinating" these hidden movements from raw audio—a process known as acoustic-to-articulatory inversion (AAI)—researchers are building systems that can better understand disordered speech, recognize emotions, and even animate digital avatars with anatomical precision.

The Inverse Problem of Speech

The fundamental challenge in this field is that the relationship between sound and movement is "one-to-many": different vocal tract configurations can sometimes produce similar sounds. Early work by Yoshiaki Shirai established that articulatory parameters could be estimated from acoustic features like formant frequencies using nonlinear regression. This approach treats the vocal tract as a physical constraint that stabilizes the highly variable acoustic signal.

Modern techniques have refined this "inversion." For instance, Olov Engwall demonstrated that combining audio with visual cues of a speaker's face allows for the reconstruction of hidden tongue movements using kernel-based regression. More recently, Shujie HU and colleagues showed that neural networks can "estimate" ultrasound tongue imaging features directly from speech, providing a high-fidelity proxy for physical sensors without requiring the actual hardware.

Why Movement Outperforms Sound

Why go through the trouble of estimating movement instead of just using the audio? The intuition is that physical gestures are more "honest" and consistent than acoustic spectra.

  1. Robustness to Variability: Acoustic signals change drastically based on background noise or a speaker's specific accent. However, the underlying articulatory goals remain relatively stable. Vikramjit Mitra found that incorporating these estimated physical constraints reduces word error rates in speech recognition by providing a representation less sensitive to acoustic "noise."
  2. Decoding Disordered Speech: For speakers with dysarthria, the acoustic output is often heavily distorted. Frank Rudzicz framed this not as "bad audio," but as a noisy-channel distortion of abstract articulatory goals. By focusing on the intended physical movements, systems can recover the speaker's meaning more accurately.
  3. Accent and Identity: Yoonjeong Lee and Shrikanth Narayanan have used AAI to show that as an accent grows stronger, specific physical gestures—like the retraction of the tongue—change in predictable ways. This physical "signature" is also useful for speaker verification, as Ming Li 0026 showed that tongue and lip trajectories capture traits that standard audio features miss.

The Kinematic Signature of Emotion

One of the most compelling applications of this research is in Speech Emotion Recognition (SER). We don't just change our pitch when we are sad or angry; we change how we move our bodies.

Donna Erickson and Caroline Menezes discovered that "real" sadness involves unique articulatory gestures—specifically higher and more forward tongue positions—that are absent in simulated or "acted" grief. This suggests that physiological arousal fundamentally shifts the spatial coordinates of speech.

Research by Sungbok Lee and Shrikanth Narayanan using real-time MRI has categorized these shifts into two broad patterns:

  • Hyper-articulation: High-arousal emotions like anger trigger faster, larger, and more forceful movements.
  • Hypo-articulation: Low-arousal states like sadness result in reduced physical activity and "lazier" tongue positions.

Interestingly, the underlying "rules" of movement remain the same. Sungbok Lee found that speech articulators follow the "one-third power law" (a mathematical relationship between speed and curvature) regardless of emotion. Emotions simply act as a "velocity gain factor," scaling the speed of the movement up or down without breaking the fundamental geometry of the gesture.

From Audio to Avatars

The ultimate test of these models is synthesis. If we can truly map audio to movement, we should be able to drive a digital character's "inner mouth" using only their voice. Tejas Prabhune and Gopala Anumanchipalli developed a streaming pipeline that does exactly this, achieving low-latency speech-to-avatar synthesis. By predicting the coordinates of the tongue and teeth in real-time, they create animations that are not just visually convincing, but anatomically accurate.

By shifting the focus from the sound wave to the physical engine that produces it, this research direction provides a more stable, biologically grounded framework for how machines process human language.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_inferring_vocal_tract_dynamics_from_spee_97fa3874,
  title = {Inferring Vocal Tract Dynamics from Speech Signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inferring-vocal-tract-dynamics-from-speech-signals/rdr_97fa387427bf47db805f326608141ddc},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_inferring_vocal_tract_dynamics_from_spee_97fa3874_v1,
  title = {Inferring Vocal Tract Dynamics from Speech Signals},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/inferring-vocal-tract-dynamics-from-speech-signals/rdr_97fa387427bf47db805f326608141ddc/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}