Acoustic-to-Articulatory Inversion Based on Speech Decomposition and Auxiliary Feature

Jianrong Wang, Jinyu Liu, Longxuan Zhao, Shanyu Wang, Ruiguo Yu, Li Liu

Introduction

Acoustic-to-articulatory inversion (AAI) is the computational process of estimating the movement of vocal articulators—such as the tongue, lips, and jaw—from a raw speech signal. This technology holds significant promise for clinical applications, including speech therapy and pronunciation training, as well as for improving robust speech recognition systems for individuals with vocal impairments.

Despite its potential, AAI faces two primary challenges. First, the speaker-independent gap arises because individual vocal tract shapes and speaking styles vary significantly; a model trained on one speaker often fails to generalize to another. Second, the information bottleneck stems from the "one-to-many" mapping problem, where different articulatory configurations can produce acoustically similar sounds, making it difficult to reconstruct precise trajectories from audio alone.

The SAFN Framework

To address these challenges, this research introduces the Speech Decomposition and Auxiliary Feature Network (SAFN). The framework is designed to decouple speaker-specific traits from linguistic content while leveraging visible articulatory cues to guide the estimation of hidden ones.

Speech Decomposition Network (SDN)

The SDN is a pre-trained module inspired by advances in speech synthesis. It utilizes self-supervised learning to decompose the input audio into two distinct representations:

  1. Speaker Embeddings: Capturing the unique "vocal thumbprint" and identity of the speaker.
  2. Content Embeddings: Capturing the underlying linguistic information.

By normalizing the speech signal in this way, the model can better handle unseen speakers by focusing on the linguistic content while accounting for individual vocal characteristics.

Auxiliary Feature Network (AFN)

The AFN addresses the information bottleneck by introducing intermediate physical constraints. While tongue movements are internal and hidden, lip movements are often highly correlated with the resulting sound and are easier to model. The AFN estimates "lip auxiliary features"—specifically the movements of the upper lip, lower lip, and lower incisors—from the audio. These serve as a structural prior, providing a physical "stepping stone" for the more complex task of tongue trajectory prediction.

Feature Transformation and Inversion

The Feature Transformation Network (FTN) fuses the personalized speech features from the SDN with the estimated lip features from the AFN. This refined data is then passed to the Articulatory Inversion Network (AIN), which employs Bi-directional Long Short-Term Memory (BLSTM) layers to predict the final, continuous trajectories of the articulators.

Experimental Setup and Results

The SAFN framework was evaluated using three benchmark datasets: MOCHA-TIMIT (two British English speakers), MNGU0 (one male British English speaker), and HPRC (Haskins Production Rate Comparison, featuring eight native American English speakers). The HPRC dataset was particularly critical for evaluating the model's performance in speaker-independent scenarios.

The results demonstrate that SAFN achieves a new state-of-the-art (SOTA) in AAI performance:

  • Speaker-Dependent Performance: The model achieved an average reduction in Root Mean Square Error (RMSE) of 0.25mm and a 2.0% increase in the Correlation Coefficient (CC) compared to previous models.
  • Speaker-Independent Performance: The improvements were more pronounced in this challenging setting, with a 0.29mm decrease in RMSE and a 5.0% increase in CC.

Ablation studies confirmed the necessity of both the SDN and AFN. Removing either module resulted in a significant performance degradation, validating the hypothesis that both speaker normalization and auxiliary physical features are essential for high-fidelity articulatory reconstruction.

Conclusion and Future Directions

The SAFN framework represents a significant step forward in making AAI systems more robust and generalizable. By explicitly modeling speaker identity and utilizing lip movements as an auxiliary guide, the system overcomes traditional barriers in speaker-independent inversion.

Future research will focus on incorporating Meta-Learning to further enhance the system's ability to adapt to new speakers with minimal data. Additionally, the authors aim to refine the self-supervised components of the network to further reduce the reliance on labeled articulatory data, which is often difficult and expensive to collect.

Summary updated

BibTeX
@inproceedings{wang2022acoustic,
  author = {Jianrong Wang and Jinyu Liu and Longxuan Zhao and Shanyu Wang and Ruiguo Yu and Li Liu},
  title = {{Acoustic-to-Articulatory Inversion Based on Speech Decomposition and Auxiliary Feature}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  doi = {10.1109/icassp43922.2022.9746125},
}
Figure