TL;DR
This paper tackles the challenge of recognizing soft whispers using throat microphones, which are less susceptible to ambient noise than traditional microphones. By exploring various adaptation techniques, the researchers significantly improved whisper recognition accuracy, demonstrating the potential of throat microphones for private and unobtrusive speech interfaces.
What Problem Does This Paper Address?
Traditional speech recognition systems rely on air-conducted sound captured by microphones. However, this method struggles with soft whispers, which are often too quiet to be reliably captured, and also picks up unwanted background noise. This limits applications requiring private communication (e.g., in banking or quiet environments) or where users want to avoid disturbing others. Throat microphones offer a promising alternative by capturing speech vibrations directly from the skin, bypassing air transmission and thus being more robust to noise. However, adapting existing speech recognition models, typically trained on normal speech from air-conducted microphones, to the unique characteristics of soft whispers captured by throat microphones is a significant challenge.
Methods
The paper explores a range of adaptation techniques to bridge the gap between standard speech recognition models and the specific data from soft whispers recorded by throat microphones. These methods can be broadly categorized:
- Maximum Likelihood Linear Regression (MLLR): This is a standard technique for adapting acoustic models. The authors investigate supervised (MLLRS), supervised+unsupervised (MLLRS-U), and a combined unsupervised approach (MLLRSU).
- Re-training with Data Transformation:
- Downsampling: The BN training data was downsampled to match the limited bandwidth of the throat microphone.
- Sigmoidal Low-Pass Filtering: A more sophisticated filter was applied to better model the frequency characteristics of the throat microphone.
- Linear Multivariate Regression (LMR): This technique was used for phone-based transformations to model the channel differences between air-conducted and throat microphones. They applied LMR at different feature extraction stages (log Mel-spectra, MFCC, CMN-MFCC).
- Feature-Space Adaptation (FSA): This method adapts the feature space itself, often used in conjunction with other techniques.
- Speaker Adaptive Training (SAT): This is a form of FSA that specifically accounts for speaker variability.
- Group MLLR and Group FSA: These approaches leverage adaptation data from multiple speakers to improve robustness, especially when individual adaptation data is scarce.
- Increased MLLR Iterations: The authors explored running MLLR for a larger number of iterations to see if further improvements could be achieved.
Datasets
- Broadcast News (BN) Data: This large dataset (66.48 hours from 6466 speakers) was used for training the initial baseline speech recognizer.
- Custom Whisper Dataset: A new dataset was collected for this research, consisting of whispered and normal speech from four native American speakers. For each speaker, 50 sentences were recorded using both a throat microphone and a close-talking microphone. The 38 phonetically-balanced sentences were used for adaptation, and the 12 sentences from news articles were used for testing.
Key Insights and Ideas
- Throat microphones are noisy for normal speech: Even with adaptation, normal speech captured by a throat microphone results in significantly higher word error rates compared to a close-talking microphone. This highlights their specialized use case for whispers.
- Simple downsampling isn't enough: While downsampling the training data to match the throat microphone's bandwidth provided some improvement, it was not sufficient to overcome the acoustic differences.
- LMR is effective for channel modeling: Linear Multivariate Regression proved to be a powerful technique for modeling the specific spectral transformations introduced by the throat microphone, especially when applied to MFCC features.
- Combining techniques yields best results: The most significant improvements were achieved by combining multiple adaptation strategies, such as LMR for channel modeling followed by speaker-dependent MLLR and FSA for further adaptation.
- More MLLR iterations help, but with diminishing returns: Increasing the number of MLLR iterations led to performance gains, but the improvements plateaued around 50 iterations.
Key Results
The paper demonstrates a dramatic reduction in word error rate (WER) for soft whisper recognition using a throat microphone. The initial baseline system achieved a WER of 99.3%. Through a series of adaptation methods, the WER was reduced to 32.9%.
Specific improvements include:
- MLLR alone reduced the WER to around 60%.
- Downsampled re-training yielded results similar to the baseline.
- Sigmoidal low-pass filtered re-training improved the WER to 53.8%.
- LMR-MFCC re-training achieved a WER of 49.8% (MLLRSU).
- Combining LMR-MFCC with FSA and SAT further reduced the WER to 40.0% (FSA-SAT, MLLRSU).
- The best performance was achieved with 50 iterations of supervised MLLR, resulting in a WER of 32.9%.
Limitations and Future Work
- Limited Adaptation Data: The authors acknowledge that the amount of adaptation data collected was small. This scarcity necessitates sophisticated adaptation techniques.
- Speaker Dependence: While group adaptation was explored, the core MLLR and FSA methods are speaker-dependent. Future work could focus on more robust speaker-independent adaptation.
- Normal Speech Performance: The paper notes that the throat microphone's performance on normal speech is significantly degraded, even with adaptation. Future research might explore hybrid systems that switch microphones based on speech intensity or context.
- Further Exploration of Group Adaptation: While group FSA/MLLR showed promise, further investigation into optimal group sizes and combinations could be beneficial.
Summary updated