TL;DR
This paper presents a novel method for estimating vocal tract shape trajectories directly from speech signals. By modifying the Kelly-Lochbaum model to include variable lip rounding and radiation impedance, and by creating a comprehensive codebook of vocal tract shapes, the authors can infer articulatory movements with promising accuracy, offering a new avenue for multimodal speech processing research.
What Problem Does This Paper Address?
Understanding how humans produce speech is a fundamental challenge in acoustics and linguistics. While models exist to simulate the vocal tract (VT) and its acoustic output, accurately inferring the dynamic shape of the VT from an audio signal remains difficult. This is particularly relevant for theories like the "motor theory of speech perception," which posits a direct link between speech perception and the motor commands used for speech production. The authors aim to bridge this gap by developing a method to extract articulatory information in real-time, creating a parallel "motoric" representation of speech that complements auditory analysis. This could enable detailed studies of speech communication and test hypotheses related to the motor theory.
Methods
The core of the method relies on a modified Kelly-Lochbaum (KL) vocal tract model. This model, a standard in speech synthesis, was enhanced in two key ways:
- Variable Length Lip Section: The lip section of the KL model can be dynamically adjusted to account for lip rounding, a crucial articulatory feature. This is implemented using a fractional delay with Lagrange interpolation.
- Lossy Lip Radiation Impedance: The model incorporates a more accurate representation of lip radiation impedance, which affects the acoustic output.
To map acoustic signals to articulatory shapes, the authors created a comprehensive articulatory codebook. This codebook was built by:
- Defining Anchor Shapes: Profiles for the eight Finnish vowels were established as "anchor points" in both articulatory and spectral domains.
- Generating Variants: These anchor shapes were then modulated using cosine-shaped "generating functions" derived from the sensitivity functions of a uniform tube. This process creates clusters of vowel variants around the anchor points, effectively filling the vowel triangle.
- Lip Rounding Variation: Four discrete values of lip rounding were applied to each shape.
- Constraint Application: Physiological constraints were applied to ensure realistic cross-sectional areas (0.03 cm² to 12 cm²).
This resulted in a codebook containing over 189,000 vocal tract shapes and their corresponding formant frequencies.
The trajectory estimation algorithm then works in several phases:
- Signal Analysis: The speech signal is segmented into 25 ms frames with 10 ms overlap and analyzed using Linear Prediction (LP) to extract the first four formant frequencies.
- Codebook Search: The extracted formant frequencies are used to search the codebook for a ranked list of candidate vocal tract shapes.
- Path Optimization: A dynamic programming approach (specifically, finding the shortest articulatory path) is used to select the most plausible sequence of vocal tract shapes across all frames, enforcing smooth transitions. The length of the lip section is included as an additional parameter in this distance calculation.
Datasets
The paper does not explicitly mention specific datasets used for training or evaluation in the traditional sense. Instead, it describes the creation of a synthetic dataset (the codebook) based on:
- Finnish Vowel Profiles: Used as anchor points.
- English Vowel Profiles (MRI data): Adapted to create the Finnish anchor shapes.
- Simulated Variations: The codebook itself, containing 189,280 vocal tract shapes and associated formant frequencies, serves as the primary "dataset" for the estimation process.
Key Insights and Ideas
- Bridging Acoustics and Articulation: The paper demonstrates a practical method for inferring articulatory movements from acoustic speech signals, directly addressing the "motor theory of speech perception." This opens doors for studying the relationship between how we perceive speech and how we produce it.
- Codebook as a Powerful Tool: The creation of a large, comprehensive codebook of vocal tract shapes and their acoustic correlates is a significant contribution. It allows for a more direct mapping from acoustics to articulation than purely analytical methods.
- Dynamic Lip Rounding is Key: Incorporating variable lip rounding significantly enhances the model's ability to capture realistic articulatory variations, especially for vowels.
- Smoothness is Essential for Realism: The use of dynamic programming to find the "shortest articulatory path" highlights the importance of smooth, physiologically plausible transitions between articulatory gestures. This is crucial for distinguishing between spectrally similar sounds.
- Potential for Multimodal Applications: The generated articulatory trajectories can serve as a valuable input for multimodal speech recognition systems, potentially improving robustness and providing richer information.
Key Results
- The developed method successfully estimates vocal tract shape trajectories from continuous speech.
- Qualitative Results:
- Figure 2 shows that without optimization, the estimated VT shapes for a vowel transition (/a/-/u/-/i/) are jerky and lack articulatory realism.
- Figure 3 demonstrates that with the optimization algorithm, the estimated trajectories are smooth and physiologically plausible, accurately reflecting the expected articulatory changes for the vowel sequence.
- Figure 4 shows that the method can differentiate spectrally similar sounds, like the glide /j/ and vowel /i/ in the Finnish word "joi," by capturing subtle differences in their articulatory shapes.
- The codebook contains 189,280 vocal tract shapes, providing a rich representation of possible articulatory configurations.
Limitations and Future Work
- Vowel-Centric Model: The current KL model is primarily designed for voiced oral sounds. The authors acknowledge that it needs improvement to handle fricatives, plosives, and nasal sounds effectively.
- Handling Unvoiced Segments: The method struggles with unvoiced segments where formant information is unreliable, leading to potential discontinuities in the estimated trajectories.
- Scaling Property Issues: The inherent scaling property of VT shapes can lead to extreme, though physiologically possible, shapes being detected. The authors suggest using a larger number of competing states or stricter constraints to mitigate this.
- Exactness vs. Distinctiveness: While the absolute exactness of the trajectories might not match MRI measurements, their distinctiveness in differentiating spectrally similar sounds is considered valuable for applications like Automatic Speech Recognition (ASR).
Future work suggested includes:
- Improving the optimization algorithm and the acoustic-to-articulatory mapping.
- Incorporating the nasal tract into the model.
- Developing methods to handle non-glottal sound sources (fricatives, plosives).
- Using the generated articulatory trajectories as an information source for multimodal speech recognition experiments.
Summary updated