TL;DR
This paper investigates how well we can predict a speaker's tongue and lip movements from their speech sound, even when the speaker's accent or language differs from the training data. They found that while speaker-independent systems can achieve moderate accuracy, cross-language and cross-accent predictions significantly reduce performance, highlighting the challenges of generalizing articulatory models.
What Problem Does This Paper Address?
Estimating articulatory movements (how the tongue, lips, and jaw move) from speech acoustics is a challenging but valuable task. It has potential applications in speech therapy, pronunciation training, and even improving automatic speech recognition. However, current "acoustic-to-articulatory inversion" systems often struggle with speaker variability, limited simultaneous acoustic and articulatory data, and inconsistent data collection methods. This leads to systems that perform well for a specific speaker but fail to generalize to new speakers, accents, or languages. This paper aims to systematically evaluate how speaker, language, and accent variations impact the performance of these inversion systems, particularly focusing on achieving speaker-independent models.
Methods
The core of this research lies in training and evaluating deep and shallow neural network architectures for speech inversion. The process involves:
- Acoustic Feature Extraction: Mel-Frequency Cepstral Coefficients (MFCCs) are extracted from the speech signal. To capture temporal context, these features are augmented with information from surrounding frames (a contextual window of 160ms).
- Neural Network Architecture: The authors employ neural networks with one to three hidden layers, each containing 300 nodes. The choice of the number of hidden layers is heuristically determined by the amount of available data.
- Output Targets: The networks are trained to predict either:
- Electromagnetic Articulography (EMA) Sensor Positions: Direct 3D coordinates of sensors placed on the articulators.
- Tract Variables (TVs): Geometrically derived measures representing constriction location and degree along the vocal tract, which are intended to be more speaker-independent than raw sensor positions.
- Normalization: Crucially, the researchers emphasize the importance of appropriate normalization. MFCCs are normalized for mean and variance per speaker. EMA sensor positions are normalized to have a mean of 0 for each recording and then speaker-wise variance normalization. This aims to mitigate speaker-specific anatomical differences.
- Smoothing: The raw output of the neural network is post-processed using a Kalman filter to smooth out high-frequency noise and obtain more stable articulatory estimates.
- Evaluation Metric: The primary evaluation metric is the Pearson product-moment correlation coefficient ($r$), which measures the linear relationship between the actual and predicted articulatory movements. Root Mean Squared Error (RMSE) is also used for EMA sensor positions.
Datasets
The study utilizes a substantial articulatory dataset collected as part of a project comparing native and non-native English pronunciation. Two subsets are specifically used:
- NL Data: Articulatory and acoustic data from 21 native Dutch speakers reading both Dutch and English words and sentences. This data is further split into:
- NL_dutch: Dutch utterances.
- NL_english: English utterances.
- NL_all: All utterances (Dutch and English).
- UK Data: Articulatory and acoustic data from 22 native British English speakers reading English words and sentences (UK_english).
This dataset is notable for its size (over 40 speakers) and the inclusion of both native and non-native speakers of English, allowing for robust cross-language and cross-accent evaluations.
Key Insights and Ideas
- Normalization is Key: The paper strongly suggests that proper normalization of both acoustic features and articulatory targets is critical for achieving reasonable speaker-independent performance. Without it, speaker variability would likely overwhelm the models.
- Tract Variables Offer Limited Advantage: While Tract Variables (TVs) are designed to be more speaker-independent than raw EMA sensor positions, the study found only a marginal improvement in prediction accuracy when using TVs compared to sensor positions. This suggests that the normalization applied to sensor data might already be effectively abstracting away from anatomical differences.
- Cross-Language/Accent Performance Drops Significantly: The most striking finding is the substantial decrease in prediction accuracy when the system is tested on data from a different language or accent than it was trained on. This highlights the inherent difficulty in building truly universal articulatory models.
- Data Quality Matters: The authors hypothesize that the UK English data, potentially collected in a more controlled environment (e.g., soundproof booth), leads to better performance when used for training compared to the NL data. This suggests that cleaner acoustic and articulatory data can lead to more robust models.
Key Results
- Speaker-Independent Performance (Matched): When trained and tested on data from the same language and accent (e.g., UK_english system on UK_english data), the speaker-independent systems achieve a correlation of approximately $r = 0.53$ for both sensor positions and TVs.
- Cross-Language Performance: When a system trained on UK English is tested on Dutch speakers' English utterances (NL_english), the correlation drops to around $r = 0.47$.
- Cross-Accent Performance: Comparing UK English speakers to Dutch speakers speaking English (NL_english vs. UK_english) shows a performance drop, though less severe than full language mismatch.
- Language Mismatch (No Data Mismatch): Comparing Dutch speakers' English utterances to Dutch speakers' Dutch utterances (NL_english vs. NL_dutch) shows a performance reduction of about 0.05 in correlation, isolating the effect of language mismatch.
- RMSE: Table 3 shows RMSE values for EMA sensor positions, indicating the magnitude of prediction errors in normalized sensor coordinates.
Limitations and Future Work
- Performance Gap: The achieved correlation of $r = 0.53$ is noted as being lower than some previous studies (e.g., $r = 0.62$ reported in [11]), though the authors argue their results are not dependent on a specific reference group.
- Sensor/Variable Influence: The prediction of certain EMA sensor positions (UL, SL) and a specific tract variable (LW) is particularly challenging and contributes to the overall lower correlation. These articulators may have a less pronounced impact on the acoustic signal.
- Future Directions:
- Speaker Normalization: The authors suggest that speaker normalization techniques could further improve performance.
- Data Combination: Combining data collected with different protocols (even across different languages) is proposed as a way to generate improved speech inversion systems.
- Cross-Protocol and Cross-Modality: Future work could explore methods for combining data from different collection protocols and even different modalities (e.g., acoustic and visual speech).
Summary updated