TL;DR
This paper introduces a novel Silent Speech Interface (SSI) that uses ultrasound tongue imaging to predict continuous vocoder parameters, including a non-discontinuous fundamental frequency (F0). This approach leads to more natural synthesized speech compared to traditional methods that rely on discrete voiced/unvoiced decisions, offering a step towards more robust and real-time SSI applications.
What Problem Does This Paper Address?
The core challenge this paper tackles is improving the naturalness and robustness of Silent Speech Interfaces (SSIs). SSIs aim to generate speech from articulatory movements, bypassing the need for actual vocalization. A key hurdle in this process is accurately modeling the acoustic speech signal, particularly the fundamental frequency (F0) and voicing information. Traditional methods often struggle with the abrupt transitions between voiced and unvoiced speech segments, leading to audible artifacts. Existing approaches that attempt to predict F0 from articulatory data have shown promise but often still produce unnatural speech due to the discrete nature of voicing decisions. This paper seeks to bridge this gap by employing a continuous vocoder that handles F0 and voicing in a more fluid manner, directly from ultrasound tongue imaging data.
Methods
The researchers propose an SSI system that leverages Convolutional Neural Networks (CNNs) to map ultrasound tongue images (UTI) to continuous vocoder parameters. The key innovation lies in the use of a continuous vocoder. Unlike traditional vocoders that make a hard voiced/unvoiced decision, this continuous vocoder models F0 as a continuous signal, even during unvoiced segments, and uses a Maximum Voiced Frequency (MVF) parameter to delineate voiced and unvoiced components.
Specifically, the system employs three speaker-dependent CNNs:
- ContF0 Prediction: Predicts the continuous fundamental frequency.
- MVF Prediction: Predicts the Maximum Voiced Frequency, which acts as a dynamic boundary for voicing.
- MGC-LSP Prediction: Predicts Mel-Generalized Cepstral coefficients, which represent the spectral envelope of the speech.
The CNNs are trained using UTI as input. The continuous F0 and MVF predictions are then used to drive the synthesis process, with MVF guiding the transition between voiced excitation (derived from a residual signal) and unvoiced excitation (white noise). This contrasts with their baseline system, which uses separate CNNs for discrete voiced/unvoiced classification and F0 regression.
Datasets
The study utilized a dataset of 209 sentences read aloud by four Hungarian speakers (two male, two female). The ultrasound tongue images were captured using the "Micro" ultrasound system at 81.67 fps, and audio was recorded simultaneously. The data was partitioned into training (85%), validation (10%), and test (5%) sets. No new benchmark datasets were introduced; the focus was on applying the proposed method to existing data.
Key Insights and Ideas
- Continuous F0 is Crucial for Naturalness: The paper strongly suggests that modeling F0 as a continuous signal, rather than a discrete voiced/unvoiced binary, significantly improves speech synthesis quality. This is because natural speech rarely has abrupt, artificial transitions.
- MVF as a Dynamic Voicing Boundary: The Maximum Voiced Frequency (MVF) acts as an intelligent switch between voiced and unvoiced excitation. This allows for smoother transitions and compensates for potential inaccuracies in the continuous F0 prediction during unvoiced segments.
- CNNs Excel at Articulatory Feature Extraction: The use of CNNs proves effective in extracting relevant articulatory features from ultrasound images for predicting acoustic parameters.
- Speaker-Dependent Models are Effective: The study demonstrates that training speaker-dependent CNN models yields good results, suggesting that individual articulatory patterns can be learned effectively.
Key Results
- Objective Evaluation: The continuous F0 prediction resulted in a significantly lower Root Mean Square Error (RMSE) of 31 Hz compared to the baseline system's 65 Hz for discontinuous F0. MVF prediction RMSE ranged from 654-1177 Hz.
- Subjective Listening Tests: While not always statistically significant, subjective tests showed a slight preference for the proposed continuous vocoder over the baseline, particularly when predicted ContF0 and MVF were used with original spectral features. The proposed system achieved an average naturalness score of 43.45% in one configuration, compared to 40.87% for the baseline. Natural sentences were rated highest (100%), and a constant F0 benchmark was rated lowest.
Limitations and Future Work
- Statistical Significance: The subjective listening tests revealed only slight, non-statistically significant improvements for the proposed system in some configurations.
- Frame Rate Constraint: The system is tied to the ultrasound frame rate (approximately 12 ms), which is slower than typical speech synthesis frame rates (5 ms). This could impact real-time performance.
- Future Work: The authors suggest that further improvements could be achieved by exploring recurrent architectures or using multiple consecutive ultrasound images as input, which were not employed in this study to prioritize real-time feasibility. They also note that the continuous vocoder's simplicity (two excitation parameters) makes it computationally feasible for real-time applications.
Summary updated