TL;DR
This paper explores building speech synthesis systems without relying on traditional, expert-defined phonetic inventories. They introduce an "emergent phone" system that learns units from acoustic data and compare it to classic and purely orthographic approaches. While promising for low-resource languages, the emergent system currently underperforms, highlighting challenges in acoustic feature representation and clustering for human perception.
What Problem Does This Paper Address?
Creating speech synthesis systems, particularly those using unit selection or parametric methods, typically requires a pre-defined inventory of phonetic units (phones). This inventory is usually built by phonologists and phonetic experts, a process that is time-consuming, resource-intensive, and potentially suboptimal. The authors propose an "emergent phone" approach, aiming to automatically construct this inventory from speech data and linguistic constraints, which could accelerate system development for new languages or accents and formalize existing heuristics.
Methods
The core of the paper is the comparison of three approaches to building speech synthesis inventories:
- Classic System: Uses a traditional, expert-defined RP (Received Pronunciation) phone set.
- Orthographic System: Uses letters of the English alphabet as units, augmented with units for common letter sequences and word boundaries. This relies heavily on the synthesis system's ability to handle pronunciation irregularities through context.
- Emergent Phone System: This novel approach uses acoustic clustering to build an inventory. Speech is first segmented into acoustically similar units using Dynamic Time Warping (DTW). This segmentation is then guided by orthographic word boundaries. Acoustic units are clustered using algorithms like x-means, aiming to create a self-organized inventory of "emergent phones." Features like voicing and syllabic position are then derived for these emergent units.
Both unit selection and parametric HTS (HMM-based Text-to-Speech) frameworks were used to build systems for each inventory type.
Datasets
The primary dataset used was 2098 utterances (approximately 5.4 hours) of studio-recorded English speech from a single female speaker with an RP accent. This data was used for training and evaluating the different synthesis systems. For evaluation, 18 sentences were selected from a larger corpus, with specific attention paid to:
- Database Coverage: Sentences were chosen to either contain words present in the initial speech database or to maximize out-of-vocabulary words.
- LTS Complexity: Sentences were categorized based on how easily their pronunciation could be predicted from their orthography (easy, medium, hard).
Key Insights and Ideas
- The "emergent phone" concept is promising but not yet a replacement: While the goal of automating inventory creation is valuable, the current emergent system did not outperform or even match the classic system in listening tests.
- Orthography is a tricky proxy for sound: The purely orthographic system performed better than the emergent system, suggesting that even with its irregularities, the direct mapping of letters to sound units (with contextual rules) is more robust than a purely acoustically derived inventory.
- Acoustic clustering needs to align with human perception: The emergent system's failure suggests that current acoustic clustering methods, even when guided by orthography, might be capturing acoustic differences that are not perceptually relevant to humans, leading to an overly large and noisy inventory.
- LTS complexity is a nuanced factor: The study found that "easy" LTS sentences didn't necessarily yield better results than "hard" ones, suggesting that factors like word frequency and hypoarticulation might play a role.
Key Results
- Listening Test Preference: A listening test revealed a strong preference for the classic phone set system. The orthographic system was preferred over the emergent phone system.
- System Performance: No significant difference was found between the HTS and Unit Selection systems themselves. However, the inventory type had a significant impact.
- Database Coverage Impact: Sentences with more words from the original database performed better, indicating that systems struggle to generalize to unseen words.
- LTS Complexity Interaction: LTS complexity interacted significantly with database coverage and system/phoneset. The "medium" LTS complexity sentences were easier than "hard" ones, but "easy" ones were not significantly better than "hard."
- Emergent System Generalization: The emergent system, particularly for unit selection, appeared to fail to generalize well outside the initial acoustic database, as suggested by its performance in out-of-database scenarios.
- Inventory Size: Both orthographic and emergent systems required significantly larger inventories (over five times larger) than the classic system.
Limitations and Future Work
- Feature Space: The authors acknowledge that the feature space used for acoustic clustering might not adequately represent human perception of phonetic differences. Future work will focus on feature extraction methods that are more aligned with human perception.
- Clustering Algorithm: The clustering process itself may not be optimal for creating perceptually relevant categories.
- Inventory Size and Sparsity: The large inventories created by the emergent and orthographic systems lead to data sparsity issues, particularly for unit selection.
- Generalization: The emergent system's poor generalization to unseen words is a significant limitation.
- Intelligibility: While not detailed, intelligibility results followed a similar pattern to MOS scores, with the emergent system performing "catastrophically."
Future work will involve exploring new feature spaces and refining clustering techniques to better capture human perceptual distinctions, aiming to make the emergent phone approach more viable.
Summary updated