Voice is our most innate communication mode, a point emphasized early on by A. Michael Noll in Natural Language Interaction with Machines: A Passing Fad? Or the Way of the Future?. However, traditional Automatic Speech Recognition (ASR) relies on a fundamentally public and fragile medium: acoustic sound waves. This reliance introduces severe social and environmental constraints.
In public spaces, users face a constant tension between convenience and privacy, often avoiding voice interfaces to prevent being overheard or causing a disturbance—a behavioral barrier documented by Maxine Eskénazi et al. in The Real Challenge 2014: Progress and Prospects. In professional environments, as shown by Nico Herbig et al. in MMPE: A Multi-Modal Interface for Post-Editing Machine Translation and Improving the Multi-Modal Post-Editing (MMPE) CAT Environment based on Professional Translators' Feedback, ambient noise and the risk of leaking sensitive data severely limit the utility of spoken commands. Furthermore, acoustic ASR completely excludes individuals with vocal impairments.
To overcome these limitations, researchers have converged on Silent Speech Interfaces (SSIs). As reviewed by Ming Zhang et al. in Perspective Chapter: Silent Speech Interface Based on Neural Activity – A Short Review, SSIs decode speech directly from neural, muscular, or physical articulatory movements, bypassing acoustic waves entirely. By capturing the internal mechanics of speech production before or during articulation, these systems preserve absolute privacy and remain entirely immune to environmental noise.
Reading the Muscles: Surface Electromyography
One primary mechanism for silent speech recognition is surface electromyography (sEMG), which measures the electrical activity of facial and neck muscles. Geoffrey S. Meltzner et al. demonstrated in Speech recognition for vocalized and subvocal modes of production using surface EMG signals from the neck and face that sEMG can accurately decode "mouthed" or unvocalized speech by capturing the activation patterns of the muscles responsible for articulation.
The core challenge of sEMG is that unvocalized speech signals differ fundamentally from audible ones. Michael Wand et al. discovered in Impact of different speaking modes on EMG-based speech recognition that speaker experience heavily dictates performance due to variations in how individuals mouth words without audio feedback.
To extract clean features from these complex, non-linear muscle signals, Michael Wand et al. introduced a Redundant Discrete Wavelet Transform (RDWT) front-end in Wavelet-based front-end for electromyographic speech recognition. Unlike traditional Fourier transforms that lose temporal resolution, wavelets capture localized, transient muscle bursts across both time and frequency scales, significantly lowering word error rates.
[Facial Muscle Movement]
│
▼ (sEMG Sensors)
[Raw Electrical Signals]
│
▼ (Redundant Discrete Wavelet Transform)
[Time-Frequency Features] ───► [Decoder / Synthesis Engine]
Going beyond text recognition, researchers have sought to reconstruct the user's voice directly from these muscle movements. Arthur R. Toth et al. in Synthesizing speech from electromyography using voice transformation techniques applied voice transformation algorithms to map EMG features to acoustic speech features. To make the output sound more natural and less robotic, Marlene Zahner et al. developed a unit selection approach in Conversion from facial myoelectric signals to speech: a unit selection approach, concatenating actual segments of recorded speech that match the predicted myoelectric profiles.
Active Sensing: Sonar, Radar, and Ultrasound
While sEMG is powerful, it requires attaching physical sensors to the face. To achieve non-intrusive or contactless interaction, alternative paradigms use wave-based active sensing to track the physical deformation of the vocal tract.
Ultrasound Imaging
Jun Cai et al. in Recognition and real time performances of a lightweight ultrasound based silent speech interface employing a language model combined vocal tract ultrasound imaging with language models. By capturing real-time video of tongue and jaw movements from beneath the chin, the system maps geometric changes directly to phonemes, enabling lightweight silent decoding.
Acoustic Sonar
Rather than specialized medical hardware, Jian Luo 0007 et al. demonstrated in End-to-end Silent Speech Recognition with Acoustic Sensing that a standard smartphone can act as an active sonar system. The device emits inaudible acoustic signals from its speakers and records the reflections via its microphone. An attention-based encoder-decoder network tracks the minute phase shifts caused by lip movements, mapping the Doppler-like shifts directly to text.
This exact principle is extended to wearable hardware in Robust Dual-Modal Speech Keyword Spotting for XR Headsets, where ultrasonic mouth tracking complements traditional audio to maintain robust keyword spotting even when a user merely mutters a command.
Radar Tracking
Taking contactless sensing further, David Ferreira et al. introduced RaSSpeR: Radar-Based Silent Speech Recognition. This approach utilizes continuous-wave radar to map vocal tract movements. Because radar waves penetrate the outer layers of skin, the system can map internal articulatory changes without requiring direct skin contact or being sensitive to external lighting conditions.
Tissue Conduction and Discrete Manual Fallbacks
Another physical workaround involves capturing sound waves before they escape into the open air. Panikos Heracleous et al. explored this in Non-audible murmur (NAM) speech recognition using a stethoscopic NAM microphone. By pressing a stethoscopic microphone against the skin of the neck, the system captures "non-audible murmurs"—low-frequency vibrations conducted directly through bodily tissue rather than the air. This allows users to whisper at near-zero acoustic volume while providing the system with a clean, noise-isolated signal.
When any form of vocal tract movement is impractical or socially unacceptable, the need for discreet interaction forces a shift toward optimized manual input methods. Kumiko Tanaka-Ishii et al. in Entering Text with a Four-Button Device and Kenneth Ward Church et al. in The Wild Thing addressed this by designing high-bandwidth text entry systems for small form factors. By utilizing predictive language models and ambiguous keypads, they offer a silent alternative to voice dictation in socially sensitive settings.
Similarly, Andrew Crossan et al. evaluated hands-free kinetic alternatives like wrist rotation in Wrist rotation for interaction in mobile contexts, mapping out the boundaries of non-verbal interaction in dynamic mobile environments when voice is entirely off the table.
Key Foundational Work
Natural Language Interaction with Machines: A Passing Fad? Or the Way of the Future? (1980) — contributed 2 concept elements to this direction.
Entering Text with a Four-Button Device (2002) — contributed 1 concept elements to this direction.