Articulatory Gestures for Speech Production and Recognition

This direction is merged into another direction: acoustic-to-articulatory inversion. Its last version stays readable and citable here.

Version 1 of 119 papers20 conceptsfrozen 2026-10-04

Traditional speech processing treats language as a sequence of discrete, symbolic units called phonemes. However, the physical reality of speech is a continuous, fluid dance of the vocal tract. When we speak, our tongue, lips, and glottis perform overlapping movements—gestures—that do not have clean boundaries.

This research direction explores the shift from acoustic-only modeling to articulatory modeling, where speech is understood through the physical motor commands and vocal tract configurations that produce it.

The Shift from Phones to Gestures

The fundamental limitation of traditional speech recognition is "coarticulation": the way the sound of a "t" changes depending on whether the next sound is an "o" or an "e." In an acoustic-only model, this requires complex statistical workarounds.

Researchers like Xiaodan Zhuang and Louis M. Goldstein argue in The entropy of the articulatory phonological code: recognizing gestures from tract variables that we should instead model speech as an ensemble of "gestural pattern vectors." In this view, a sound is not a point in an audio spectrum, but a set of physical targets, such as "lip closure" or "tongue tip constriction." Because these physical gestures have inherent duration and can overlap in time, they naturally account for the fluid nature of speech.

The Acoustic-to-Articulatory Inversion Problem

If articulatory gestures are a more robust representation of speech, why don't we use them for everything? The challenge lies in the inverse problem: while it is easy to predict what a sound will be if you know the tongue's position, it is incredibly difficult to determine the tongue's position just by listening to the sound.

As Yasuhiro Wada and Mitsuo Kawato highlight in A Computational Model for Cursive Handwriting Based on the Minimization Principle, the brain likely solves this by optimizing for smoothness and physical efficiency (the "minimum torque-change criterion"). For machines, however, mapping high-dimensional acoustic signals back to specific motor commands remains a significant hurdle. These articulatory states are "latent variables"—hidden factors that influence the output but cannot be directly observed in the audio stream.

Bridging the Gap with Multimodal Data

To solve this inverse problem, researchers are increasingly using sensors to "see" the vocal tract in action. By providing the model with ground-truth physical data, they can train systems to recognize the relationship between movement and sound.

  • Real-time MRI: Yaoyao Yue and colleagues in Towards Speech Classification from Acoustic and Vocal Tract data in Real-time MRI used Transformer architectures to fuse audio with rtMRI video of the vocal tract. This allows the model to learn the temporal overlap of speech gestures directly from visual evidence.
  • Electropalatography (EPG): Li‐Chin Chen developed the EPG2S system, which uses sensors on the roof of the mouth to track tongue contact. This physical data allows for speech generation and enhancement that is immune to external acoustic noise.

Modeling Temporal Dynamics

Once we treat speech as a physical process, we need a way to model its timing. Unlike rigid digital signals, human movement is variable. Chi Hu and Mark Hasegawa-Johnson addressed this in FSM-based pronunciation modeling using articulatory phonological code by using Finite State Machines (FSMs).

By representing speech as a sequence of overlapping gestures with flexible timing, these models can capture the "stiffness" or "constriction degree" of the vocal tract. This approach moves away from trying to find the "right" sound and instead focuses on reaching the "right" physical target.

Biological Foundations

This research mirrors how biological systems learn. In How Perception Guides Production in Birdsong Learning, Christopher L. Fry describes how songbirds transition from hearing a tutor to producing their own song. The bird's brain creates internal perceptual categories that act as a "teacher" for its motor system. This suggests that the link between perception (hearing) and production (moving) is not just a technical convenience for engineers, but a fundamental requirement for vocal learning in nature.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_articulatory_gestures_for_speech_product_3f5fe250,
  title = {Articulatory Gestures for Speech Production and Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/articulatory-gestures-for-speech-production-and-recognition/rdr_3f5fe2504f8b4c35809e88612cac09ba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_articulatory_gestures_for_speech_product_3f5fe250_v1,
  title = {Articulatory Gestures for Speech Production and Recognition},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/articulatory-gestures-for-speech-production-and-recognition/rdr_3f5fe2504f8b4c35809e88612cac09ba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}