Abstract
This study enhances the exemplar-based Context Sequence Model (CSM) by incorporating raw articulatory data—specifically physical movements of the vocal tract—alongside traditional acoustic signals. Using a Polish speech corpus, the researchers demonstrate that raw, continuous articulatory trajectories can serve as robust memory representations for segment selection in speech production. The findings suggest that motor-like habits can be directly leveraged in memory-based synthesis, with articulatory models performing comparably to or better than acoustic-only counterparts.
Introduction
Traditional models of speech production often rely on abstract phonological rules or purely acoustic representations. However, "Exemplar Theory" suggests that human speech perception and production rely on a rich memory bank of highly detailed, previously encountered speech instances (exemplars). The Context Sequence Model (CSM) is an existing exemplar-based model that produces speech segment-by-segment by matching the current acoustic context to stored memories.
Since speech is fundamentally a physical, motor-driven process, this paper addresses whether raw articulatory features—the actual movements of the lips, tongue, and jaw—can be integrated into the CSM. The authors investigate if this physical context improves the model's ability to select correct phonetic segments during production.
Methodology
The researchers modified the CSM to operate on both acoustic and articulatory data:
- Articulatory Tracking: Continuous 2D Electromagnetic Articulography (EMA) signals were used to track the physical positions of the vocal tract at 250 Hz.
- Exemplar Matching: To simulate production, the model uses a "leave-one-out" approach, searching its memory for the best matching exemplar based on a target sequence of phonetic labels.
- Context Restriction: To avoid bias from predictable carrier phrases, the model was restricted to using only 0.5 seconds of the preceding raw signal (acoustic, EMA, or combined) to select the next segment, stripping away upcoming linguistic context.
The Polish C-Center Corpus
The study utilized the Polish C-Center Corpus, featuring three native Polish speakers. The dataset contains 336 utterances with target words featuring simple and complex consonant onsets and codas.
Results and Analysis
The results indicate that raw articulatory data is a robust predictor for speech segment selection. All tested models (EMA-only, Acoustic-only, and Combined) significantly outperformed a random selection baseline, which would yield a 50% error rate.
EMA vs. Acoustics
For two out of the three speakers, the model using only EMA articulatory signals made fewer errors than the model using only acoustic envelope signals. This suggests that raw movement trajectories are sufficiently robust to serve directly as memory representations without complex transformations.
Error Patterns
The model demonstrated high accuracy in distinguishing syllable positions (onset vs. coda), making almost zero errors in this category across all speakers. However, it faced more challenges distinguishing between syllable types (e.g., simple consonants vs. complex clusters). Specifically, the model struggled with Polish codas, which are known in linguistic theory to be highly variable and lack the tight articulatory coupling seen in syllable onsets.
Conclusion
This research proves that motor-like habits can be directly leveraged in memory-based speech synthesis. While the performance differences between modalities were small, the robustness of raw EMA data is promising. Future work will involve scaling to larger corpora and investigating specific acoustic and articulatory "landmarks" to further refine the model's predictive capabilities.
Summary updated