TL;DR
This paper introduces a novel silent speech interface that uses magnetic sensors to detect the movement of small magnetic pellets attached to speech articulators. The system achieves impressive word recognition accuracy (up to 98.7%) on a small vocabulary, demonstrating its potential as an alternative communication method for individuals unable to speak conventionally.
What Problem Does This Paper Address?
The paper addresses the challenge of enabling communication for individuals who have lost their ability to vocalize, such as those who have undergone laryngectomy or have conditions affecting their vocal cords. Traditional speech recognition relies on acoustic signals, which are unavailable in these cases. The authors propose a silent speech interface that captures articulatory movements, offering a potential solution for these individuals and for applications requiring non-acoustic communication in extreme environments.
Methods
The core of the REDRESS silent speech interface involves attaching small magnetic pellets to key speech articulators (lips and tongue). A wearable frame, similar to a headset, houses magnetic field sensors that detect changes in the magnetic field induced by the pellets as the articulators move during silent speech. Instead of directly translating these changes into precise 3D positions, the system uses the raw magnetic field data (and its derivatives) as features for a Hidden Markov Model (HMM) based speech recognizer. This approach bypasses the need for acoustic signals entirely. The system was evaluated using HMMs trained on both the magnetic sensor data and, for comparison, Mel-frequency cepstral coefficients (MFCCs) from acoustic speech.
Datasets
The experiments were conducted using a custom dataset collected from a single male native English speaker. The dataset consists of 1138 utterances, covering a vocabulary of 57 words. This vocabulary includes digits and a set of words designed to cover the ARPAbet phonetic inventory, with a focus on minimal pairs to increase the difficulty of the recognition task. The data was recorded simultaneously for both magnetic sensor signals and acoustic speech.
Key Insights and Ideas
- Magnetic Sensing for Articulation: The paper demonstrates that magnetic field changes, rather than precise 3D positions of articulators, can be effectively used as features for speech recognition. This simplifies the system and potentially makes it more robust.
- Velocity Features are Key: Surprisingly, using only the first time derivatives (velocities) of the magnetic sensor data (the "SensorDex" condition) yielded the highest accuracy, even surpassing the acoustic baseline in some aspects. This suggests that the speed of articulation is a more discriminative feature than static positions or accelerations for this interface.
- Potential for Minimal Pair Discrimination: While minimal pairs remain challenging, the use of velocity features significantly improved discrimination for some pairs (e.g., bond/pond), indicating that the system can capture subtle articulatory dynamics.
- Promising for Small Vocabularies: Even in its early stages, the system shows strong performance on a 57-word vocabulary, suggesting a clear path towards practical applications for command-and-control or communication aids.
Key Results
The REDRESS silent speech interface achieved a word accuracy of 98.7% in the "SensorDex" condition (using velocity features of magnetic sensor data). This is remarkably close to the performance of the acoustic baseline using MFCCs, which achieved 99.7% accuracy. The "Sensor" condition (raw magnetic data) achieved 88.5% accuracy, highlighting the importance of feature extraction. Adding accelerations ("SensorDD") did not improve performance over just velocities ("SensorD").
Limitations and Future Work
The authors acknowledge several limitations and outline future directions:
- Single Speaker: The current evaluation was performed on a single speaker, and future work will involve testing on multiple speakers, including medical trials with patients.
- Auditory Feedback: The test speaker had auditory feedback, which may have influenced performance. Future trials will need to evaluate performance without this feedback, as would be the case for laryngectomy patients.
- Earth's Magnetic Field: The influence of the Earth's magnetic field was addressed through background cancellation, but the authors propose using reference sensors for more robust cancellation in future versions.
- Pellet Placement and Optimization: Further optimization of magnetic pellet placement and the derivation of more statistically suitable feature vectors are planned.
- Velum Movement: The current system does not capture velum movement, which could be incorporated by adding more pellets.
- Large Vocabulary Recognition: While promising for small vocabularies, extending the system to large vocabulary speech recognition is a future goal.
Summary updated