Technologies for processing body-conducted speech detected with non-audible murmur microphone

Tomoki Toda, Keigo Nakamura, Takayuki Nagai, Tomomi Kaino, Yoshitaka Nakajima, Kiyohiro ShikanoInterspeech 2009

TL;DR

This paper explores technologies for processing "body-conducted speech" captured by a novel Non-Audible Murmur (NAM) microphone. They've developed methods for both converting this speech to be more natural-sounding for human-to-human communication and for recognizing it for man-machine interaction, demonstrating promising results for more universal and robust speech interfaces.

What Problem Does This Paper Address?

Traditional speech communication relies on audible sound waves, which can be problematic in various situations. Imagine trying to have a private conversation in a crowded public space, speaking in a quiet library, or for individuals who have undergone surgery affecting their vocal cords (like a laryngectomy). These scenarios highlight limitations in current speech technology. The paper addresses the need for speech communication methods that are more versatile, private, and accessible, particularly for those with speech impairments or in noisy environments. They aim to overcome these barriers by leveraging body-conducted speech, which travels through tissues rather than air.

Methods

The core of this research revolves around the Non-Audible Murmur (NAM) microphone. This device, inspired by a stethoscope, is placed on the neck and captures subtle vibrations from the vocal tract, even when speech is extremely soft or silent. Its noise-proof structure makes it inherently robust against external noise.

The paper details two main technological thrusts:

  • Body-Conducted Speech Conversion: This aims to improve the quality of NAM-captured speech for human-to-human communication. They employ statistical voice conversion techniques, specifically using Gaussian Mixture Models (GMMs). The goal is to transform the degraded body-conducted speech into more natural-sounding speech, either normal speech or whisper, depending on the desired application (e.g., silent communication vs. noise-robust communication). They also propose a novel speaking aid system for laryngectomees that combines low-power excitation signal generation, NAM detection, and voice conversion to whisper.
  • Body-Conducted Speech Recognition: This focuses on enabling machines to understand NAM-captured speech, making man-machine interfaces more universal. The primary challenge here is building acoustic models specific to body-conducted speech, as it differs significantly from air-conducted speech. They utilize adaptation techniques like Maximum Likelihood Linear Regression (MLLR) to adapt existing normal speech models to NAM data. They also explore training speaker-independent NAM models and using speaker adaptive training (SAT) to improve recognition accuracy for general speakers and various speaking styles.

Datasets

The paper describes experiments using datasets collected with the NAM microphone.

  • General Speaker NAM Recognition: Data was collected from 58 general speakers uttering NAM. This data was used for training and adaptation. A standard Japanese newspaper article corpus was used for training the language model.
  • Various Speaking Styles Recognition: Data was collected from one female speaker in four speaking styles: NAM, whisper, soft voice, and normal speech, under both clean and noisy conditions (50, 60, and 70 dBA office noise). Voiced and unvoiced speech were considered under noisy conditions.

Key Insights and Ideas

  • NAM Microphone's Versatility: The NAM microphone isn't just for silent speech; it can capture a wide range of body-conducted speech, from extremely soft murmurs to normal speech, making it suitable for diverse communication scenarios.
  • Body-Conducted Speech Quality Degradation: The natural transmission of sound through soft tissue leads to inherent quality issues (e.g., lack of lip radiation, low-pass filtering effects) that require specific processing.
  • Voice Conversion for Quality Enhancement: Statistical voice conversion is effective in improving the intelligibility and naturalness of body-conducted speech, especially when converting to a target style that matches the input's characteristics (e.g., converting soft voice to soft voice).
  • Speaker Adaptation is Crucial for General Users: Initial attempts at NAM recognition worked well for trained speakers, but for general users, speaker-independent models and speaker adaptive training (SAT) are essential for achieving acceptable accuracy.
  • Noise Robustness is a Key Advantage: The inherent noise-proofing of the NAM microphone and the nature of body conduction offer significant advantages for speech recognition in noisy environments.
  • Style-Mixed vs. Style-Dependent Models: For recognizing various speaking styles, style-dependent models generally outperform style-mixed models, especially under noisy conditions, though parallel decoding with style-dependent models offers a practical solution.

Key Results

  • General Speaker NAM Recognition: Using speaker-adaptive training (SAT) with NAM data improved word accuracy from 64.43% (conventional SI-Normal) to 72.58% (SAT-SI-Normal), significantly reducing inter-speaker variation.
  • Speaking Style Recognition (Clean Conditions): Matched style-dependent models achieved the highest word accuracies, for example, 89.41% for Normal speech and 77.90% for NAM. Parallel decoding results were close to matched models.
  • Speaking Style Recognition (Noisy Conditions): Similar trends were observed, with matched models performing best. For instance, under 50 dBA noise, accuracies reached 88.22% for voiced speech and 81.84% for unvoiced speech. The paper notes that 60 dBA noise sometimes led to worse performance, possibly due to less steady speech production.

Limitations and Future Work

  • Inter-Speaker Variation: While SAT improved accuracy, the variation in recognition performance across different speakers remains significant, indicating a need for further research to reduce this variability.
  • Noisy Condition Performance: Performance degradation in noisy conditions, particularly for unvoiced speech, suggests that more advanced noise-robust techniques are needed.
  • Limited Speaker Data for Styles: The evaluation of various speaking styles was conducted with only one female speaker, limiting the generalizability of these findings. Future work should involve collecting data from more speakers across different styles and noise levels.
  • Lombard Reflex Impact: The paper briefly mentions the negative impact of the Lombard reflex (involuntary speech changes in noisy environments) on NAM recognition and suggests further investigation into coping with this.
  • Further Model Development: The authors indicate that further refinement of acoustic models and adaptation techniques is necessary for more robust and accurate body-conducted speech recognition.

Summary updated

BibTeX
@inproceedings{toda2009technologies,
  author = {Tomoki Toda and Keigo Nakamura and Takayuki Nagai and Tomomi Kaino and Yoshitaka Nakajima and Kiyohiro Shikano},
  title = {{Technologies for processing body-conducted speech detected with non-audible murmur microphone}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure