Introduction
Acoustic-to-articulatory inversion (AAI) is the process of estimating the physical movements of speech articulators—such as the tongue, lips, and jaw—from an acoustic speech signal. This technology holds significant promise for applications in speech therapy, pronunciation training for second-language learners, and the enhancement of robust speech recognition systems.
Despite its potential, AAI faces two primary obstacles: the extreme difficulty of collecting high-fidelity articulatory data (typically requiring electromagnetic articulography) and the challenge of speaker variability. Because speech signals contain a mixture of linguistic content and speaker-specific identity, traditional models often struggle to generalize to new, unseen speakers. This paper introduces the Speech and Phoneme stream Network (SPN), a dual-pathway architecture designed to overcome these hurdles by decoupling linguistic content from acoustic texture.
Methodology: The Two-Stream Approach
The SPN model utilizes a two-stream joint-training framework to capture both the local acoustic nuances and the global linguistic context of speech.
1. The Speech Stream
This pathway processes Mel-frequency cepstral coefficients (MFCCs) to extract acoustic features. While traditional models rely heavily on 1D Convolutional Neural Networks (1DCNNs) to capture local temporal patterns, the SPN enhances this stream with a Multi-head Attention module. This addition allows the model to capture "global" dependencies across distant frames, providing a more holistic representation of the speech segment.
2. The Phoneme Stream
The phoneme stream serves as the foundation for speaker independence. By processing transcribed phonemes through a Bidirectional Long Short-Term Memory (BLSTM) network, the model gains access to a representation of speech that is inherently independent of individual vocal characteristics. Phonemes represent the underlying "mechanical intent" of the speaker, providing a stable anchor that guides the articulatory prediction regardless of the speaker's identity.
3. Joint Training and Fusion
Rather than using a pre-trained, static phoneme model, the SPN employs joint training. Features from both the speech and phoneme streams are fused and fed into an Articulatory Inversion Network (AIN). This synergy allows the two streams to optimize simultaneously, ensuring that the acoustic features and linguistic content complement one another during the final trajectory estimation.
Experimental Setup
The model was evaluated using the HPRC (Haskins) dataset, which consists of 720 utterances from each of eight native American English speakers (four male, four female). The task involved predicting the X and Z coordinates for six key articulatory points: the tongue (T1, T2, T3), upper lip (UL), lower lip (LL), and lower incisor (LI).
To rigorously test speaker independence, the researchers employed a "leave-one-speaker-out" cross-validation strategy. This ensures that the model is always evaluated on a speaker whose voice was not present during the training phase.
Results and Discussion
The SPN model demonstrated a significant advancement over previous state-of-the-art (SOTA) methods in speaker-independent scenarios:
- Error Reduction: The model achieved a Root Mean Square Error (RMSE) of 2.537mm, marking a reduction of 0.184mm compared to prior benchmarks.
- Correlation Improvement: The Pearson Correlation Coefficient (PCC) reached 0.810, a 6% improvement over previous SOTA results.
- Qualitative Fidelity: Analysis of predicted trajectories indicates that the SPN more accurately tracks complex tongue movements, closely mirroring ground-truth physical recordings where previous models often faltered.
These results suggest that the inclusion of a dedicated phoneme stream effectively mitigates the "distraction" caused by speaker-specific acoustic features, allowing the model to focus on the universal physical mechanics of speech production.
Conclusion and Future Work
The SPN architecture proves that joint training of acoustic and phonemic streams is a highly effective strategy for speaker-independent AAI. By balancing local acoustic details with global linguistic context, the model achieves superior accuracy and correlation.
However, the current reliance on manual phoneme transcriptions (via forced alignment) remains a limitation for real-time deployment. Future research will focus on integrating automated phoneme recognition to enable "in-the-wild" applications and exploring meta-learning techniques to further improve adaptation to new speakers with minimal data.
Summary updated