TL;DR
This paper introduces a novel approach to "silent speech interfaces" (SSIs) by improving the accuracy of decoding speech from visual cues of tongue and lip movements. By employing context-dependent models and a multi-stream approach to combine ultrasound and optical data, they achieve a significant performance boost over baseline methods, paving the way for more robust silent communication technologies.
What Problem Does This Paper Address?
The ability to communicate silently, without vocalizing, holds immense potential for various applications, from preserving privacy and enabling communication in noisy environments to assisting individuals with speech impairments. However, building effective Silent Speech Interfaces (SSIs) is challenging. A key hurdle is accurately decoding speech from visual information alone, as visual cues from the articulators (tongue, lips) are inherently ambiguous and highly dependent on surrounding sounds (co-articulation). This paper tackles the problem of improving the phonetic decoding stage of an SSI that relies on ultrasound and optical imaging of the vocal tract and lips, aiming to make the visual-to-phonetic mapping more robust and accurate.
Methods
The authors propose two main improvements over their previous work:
- Context-Dependent Modeling: Instead of treating each phoneme in isolation, they introduce context-dependent models. This means the model considers the preceding and succeeding phonemes (triphones) when decoding. To handle the data sparsity inherent in triphone modeling, they employ a tree-based state-tying strategy. This approach clusters similar states across different triphones, allowing for more robust model training even with limited data. They also explore using articulatory features to build these contextual questions within the decision trees.
- Multi-Stream Hidden Markov Models (MSHMMs): They move from a "feature fusion" strategy (combining visual features before modeling) to a "multi-stream" approach. In MSHMMs, each data stream (ultrasound and optical) has its own acoustic model within each state of the HMM. This allows the model to learn the distinct characteristics and emission probabilities of each stream independently before combining them. They found that the ultrasound stream carried more information, assigning it a higher weight (0.7 vs. 0.3 for optical).
These new models are evaluated against a baseline system that uses context-independent models and a feature fusion strategy.
Datasets
The study utilizes a custom-recorded, one-hour, phonetically balanced English speech database. This database consists of:
- Ultrasound recordings of the vocal tract.
- Optical recordings of the speaker's lips (frontal view).
- Synchronized audio recordings.
The speech material is based on the first 1020 sentences from the CMU Arctic corpus, read by a native American English speaker. The database contains approximately 61 minutes of speech and 109,553 frames. For evaluation, a jackknife (leave-one-out) technique is employed, dividing the data into 34 test lists.
Key Insights and Ideas
- Context is King (for Articulation): The visual cues from the tongue and lips are heavily influenced by surrounding sounds. Modeling these contextual dependencies significantly improves the accuracy of phonetic decoding, highlighting the importance of co-articulation in visual speech.
- Separate Streams, Better Understanding: Treating the ultrasound and optical data streams as independent entities within a multi-stream HMM framework allows for a more nuanced understanding of the articulatory information each provides, leading to better overall performance than simply merging features upfront.
- Ultrasound Dominance: The ultrasound data, capturing internal tongue movements, appears to be the primary source of information for phonetic decoding in this setup, carrying more weight in the multi-stream model.
- Linguistic Constraints are Crucial: While visual decoding alone is improving, the paper emphasizes that linguistic constraints (like a limited vocabulary) are still vital for achieving high accuracy, suggesting that visual information alone cannot fully disambiguate all phonetic possibilities.
Key Results
The paper presents a clear performance improvement across different evaluation scenarios:
- Baseline Decoder (Context-Independent, Feature Fusion): Achieved 57.7% accuracy in the unconstrained scenario and 67.4% in the constrained (2.5k word dictionary) scenario.
- Context-Dependent Decoder (Feature Fusion): Improved accuracy to 62.6% (unconstrained) and 70.6% (constrained).
- Multi-Stream Decoder (Context-Independent): Showed gains to 59.5% (unconstrained) and 71.1% (constrained).
- Final System (Context-Dependent, Multi-Stream): This combined approach yielded the best results, achieving 65.6% accuracy in the unconstrained scenario and 74.7% in the constrained scenario. This represents an absolute performance improvement of 8% over the baseline system in the constrained setting.
Limitations and Future Work
- Frame Rate: The current acquisition system is limited to 29.97 Hz (NTSC format). The authors suggest that recording visual data at a higher frame rate could further reduce errors, particularly deletions and insertions.
- Statistical Language Model: While a limited vocabulary dictionary was used for constrained decoding, the paper notes that the absence of a statistical language model is a limitation. Incorporating one is suggested as a future direction to further improve performance.
- Ambiguous Phones: The authors acknowledge that many remaining substitution errors occur on phones with similar tongue and lip gestures, and also with vowels and certain consonants that are difficult to image clearly with ultrasound (e.g., due to acoustic shadows).
- Psychoacoustic Effects: They speculate that psychoacoustic effects might help mitigate some unintelligible synthesis issues arising from phonetic decoding errors.
Future work will focus on addressing these limitations, particularly by increasing the frame rate of visual data acquisition and integrating statistical language models.
Summary updated