When we watch someone speak, we only see the surface of a highly complex physical process. We see the lips part, the jaw drop, and the cheeks flex. However, the primary machinery of speech production—the tongue, the soft palate (velum), the pharynx, and the larynx—operates entirely within the dark, enclosed cavity of the vocal tract. This physical barrier creates a challenge known as internal articulatory occlusion.
Because these critical articulators are physically obscured by the jaw and cheeks, speech scientists, language learners, and machine learning models face a severe visibility problem. To overcome this information bottleneck, researchers are developing imaging techniques and computational models to uncover, reconstruct, and visualize these hidden oral gestures.
The Hidden Mechanics of Speech
Traditional visual speech analysis is heavily restricted to surface-level lip movements. As A. Turkmani et al. demonstrate in Visual analysis of lip coarticulation in VCV utterances, focusing solely on the lips ignores the internal articulators that are absolutely essential for producing non-labial consonants.
When we look deeper into the vocal tract, we find that many of the most distinctive features of speech rely on completely invisible movements:
- Pharyngeal Constrictions: To produce the English rhotic ("r") sound, speakers must execute a precise constriction deep in the lower throat. Sarah Harper et al. found in L2 Acquisition and Production of the English Rhotic Pharyngeal Gesture that second-language learners frequently omit or misarticulate this gesture simply because they can neither see it nor easily feel it.
- Velic Coordination: The rapid, subtle movements of the velum (soft palate) regulate the flow of air through the nasal cavity. The authors of Velic coordination in French nasals: a real-time magnetic resonance imaging study showed that mastering the precise timing between the tongue and the velum is what differentiates native phonetic patterns from non-native ones.
- Laryngeal Control: In advanced vocal performances like human beatboxing, the secret to producing high-pressure, punchy sounds lies in rapid glottalic airstream mechanisms hidden deep within the larynx, as documented by Nimisha Patil in Comparison of Basic Beatboxing Articulations Between Expert and Novice Artists Using Real-Time Magnetic Resonance Imaging.
Even during moments of complete silence, the vocal tract is often actively moving. Christina Hagedorn et al. discovered in Characterizing Covert Articulation in Apraxic Speech Using real-time MRI that individuals with speech apraxia frequently perform "covert articulations"—silent, unvoiced movements of the tongue and lips where the speaker is physically rehearsing a sound. Without a way to peer past the cheeks, these critical motor-planning behaviors remain entirely invisible to both clinicians and acoustic analysis tools.
Peering Inside the Vocal Tract
To capture the precise geometric motion of these occluded structures, researchers utilize advanced, non-invasive imaging technologies. Ioannis Douros developed a comprehensive multi-modal database, detailed in A Multimodal Real-Time MRI Articulatory Corpus of French for Speech Research, to map these internal dynamics.
The primary tool for this work is real-time Magnetic Resonance Imaging (rtMRI). Colin Vaz leveraged high-speed rtMRI in Illustrating the Production of the International Phonetic Alphabet Sounds Using Fast Real-Time Magnetic Resonance Imaging to capture the entire vocal tract in motion at over 83 frames per second, creating a visual library of how humans coordinate the velum, pharynx, and larynx for diverse phonetic sounds. This approach has even been used to isolate the specific supraglottal configurations that give rise to perceived voice qualities, such as character archetypes in media (Emi Zuiki Murano et al. in Articulatory correlates of voice qualities of god guys and bad guys in Japanese anime: an MRI study).
Beyond MRI, Ultrasound Tongue Imaging (UTI) offers a portable way to track the tongue's contour. By synchronizing ultrasound transducers with external video and acoustic sensors, platforms like the one built by Patrick Chawah et al. in An educational platform to capture, visualize and analyze rare singing can document the hidden physical mechanics of endangered singing traditions and human beatboxing.
In clinical settings, combining these visual tracking methods with electromyography (EMG) allows physicians to uncover structural anomalies—such as hidden scar tissue bridges—that are otherwise masked by concurrent vocal fold paralysis (Mikikazu Yamagiwa et al. in A Case of Inter-Arytenoid Band).
Computational Reconstruction and Inversion
While rtMRI and ultrasound provide excellent ground-truth data, they are too expensive and cumbersome for daily use. To solve this, researchers use machine learning to reconstruct hidden internal movements from easily accessible external signals, a process known as Acoustic-to-Articulatory Inversion (AAI).
The intuition behind AAI is that the shape of the vocal tract acts as a physical acoustic filter; therefore, the resulting audio contains a mathematical signature of the internal geometry that created it. C. Siddarth et al. demonstrated this in Watch Me Speak: 2D Visualization of Human Mouth during Speech by training a deep learning model to map real-time acoustic signals directly to a 2D "X-ray" animation of the tongue, jaw, and uvula.
Similarly, Olov Engwall et al. showed in Reconstructing tongue movements from audio and video that combining standard facial video with audio allows non-linear regression models to accurately estimate hidden tongue positions without requiring physical motion-capture markers.
[Acoustic Audio + Visible Lips] ---> [Deep Neural Network] ---> [Inferred Tongue & Velum Positions]
This cross-modal mapping is particularly valuable for audio-visual speech enhancement. Because certain consonants depend entirely on hidden tongue-to-palate contact, systems that only look at the lips lack the information needed to accurately reconstruct those phonemes. To bridge this gap, Rui-Chen Zheng et al. introduced a "lip-tongue memory network" in Incorporating Ultrasound Tongue Images for Audio-Visual Speech Enhancement. During training, the model learns a shared representation between visible lip videos and internal ultrasound tongue images. During real-world inference, the model can "recall" the missing tongue features using only the visible lip movements, significantly boosting speech intelligibility.
The Cognitive Bottleneck of Visual Feedback
If we can successfully reconstruct and display these hidden movements, can we use them to improve human learning? This is the core premise of modern pronunciation training systems, such as the one introduced by Haruko Miyakoda in Development of a pronunciation training system based on auditory-visual elements, which provides dynamic, multi-angle visualizations of internal speech organs to help language learners diagnose and correct subtle articulatory errors.
However, designing effective visual feedback reveals a surprising cognitive bottleneck: human brains are not naturally wired to interpret the internal mechanics of the mouth in real time.
Preben Wik et al. explored this limitation in Can visualization of internal articulators support speech perception?. They provided listeners with a transparent, animated side-view of a synthetic face showing the tongue's movements to see if it would improve speech perception for degraded audio signals. Surprisingly, the addition of the internal visualization did not significantly improve word recognition compared to standard lip-reading cues alone.
Because we never see internal articulators during natural communication, translating an abstract, transparent side-view of a moving tongue into meaningful phonetic information requires significant cognitive effort and training. Just as educators struggle to directly observe a student's internal mental processing failures during listening drills (Yingxiang Gao et al. in Automatic Prediction of Language Learners' Listenability Using Speech and Text Features Extracted from Listening Drills), speech learners must be taught how to read these unfamiliar internal movements before the visualizations can effectively guide their own pronunciation.
Key Foundational Work
Articulatory correlates of voice qualities of god guys and bad guys in Japanese anime: an MRI study (2004) — contributed 1 concept elements to this direction.
A Case of Inter-Arytenoid Band (1980) — contributed 1 concept elements to this direction.