TL;DR
This paper tackles the challenge of speech synthesis for languages with no existing transcribed data. They propose a novel method to automatically discover "perceptual acoustic units" (AUs) from raw audio, modeling speech as transient and steady-state regions. This approach enables the creation of zero-resource speech synthesis systems that achieve good quality with reduced data requirements.
What Problem Does This Paper Address?
Speech synthesis, particularly text-to-speech (TTS), typically relies heavily on large amounts of transcribed audio data. This poses a significant barrier for languages with limited or no existing linguistic resources (e.g., low-resource languages, or languages without a written script). The problem is how to build effective speech synthesis systems when you only have access to raw audio recordings, without any corresponding text transcriptions. This is known as the "zero-resource" speech processing problem. Existing methods often struggle to discover meaningful acoustic units that can generalize well for synthesis without explicit linguistic guidance.
Methods
The core of this paper is the acoustic unit discovery (AUD) process. Instead of relying on traditional phonetic units, they aim to discover "perceptual acoustic units" (AUs) that better reflect how humans perceive speech.
Here's a breakdown of their approach:
- Syllable-like Segmentation: The speech signal is first segmented into syllable-like units using energy contours. This is a more intuitive starting point than fixed-length frames.
- Clustering of Segments: These syllable-like segments are then clustered using dynamic time warping (DTW) to measure similarity. A k-nearest neighbor connected component clustering approach is employed.
- Modeling Transient and Steady-State Regions: Crucially, they model AUs as representing transient (onset/decay) and steady-state (vowel/sonorant) regions within these segments. This is inspired by the structure of syllables (CV, VC units).
- Two-Stage HMM-GMM Training:
- Stage 1 (CVC Initialization): Initial acoustic models are trained on these clustered segments, assuming each segment contains an onset, rhyme, and offset. This is done using a self-training approach where models are iteratively refined.
- Stage 2 (Continuous Speech Training): The models from Stage 1 are then used to transcribe continuous speech, allowing for the modeling of inter-segment transients. This process is repeated to obtain final models.
- Unit Merging: To reduce the number of units and potentially improve efficiency, they merge the discovered AUs (e.g., to 40 units) and retrain the models.
Datasets
The experiments were conducted using the ZeroSpeech 2019 challenge dataset.
- Development Data: English language, with 1 male speaker and 100 utterances (approx. 15 hours).
- Test Data: Indonesian language, with 1 female speaker and 112 utterances (approx. 15 hours).
- Train Unit Dataset: Provided for AUD and acoustic modeling.
- Test Corpus: Used for synthesis, requiring the synthesized audio to be in the target speaker's voice. The speakers in the source and test data are disjoint.
The paper notes that the ZeroSpeech challenge focuses on zero-resource speech processing, with recent challenges emphasizing unit discovery and speech synthesis.
Key Insights and Ideas
- Perceptual Units Matter: The core idea is that modeling speech as perceptual units (transient and steady-state regions) is more effective for synthesis than arbitrary frame-based or even purely syllabic units, especially in a zero-resource setting.
- Transient and Steady-State Modeling: Explicitly modeling both the dynamic transitions (transients) and stable portions (steady-states) of speech segments captures richer acoustic information.
- Self-Training for Unsupervised Learning: The use of self-training is a powerful technique to iteratively refine acoustic models and transcriptions in an unsupervised manner, leveraging the initial discovered units.
- Children's Language Acquisition Analogy: The two-stage training process, starting with smaller units and then moving to continuous speech, is likened to how children learn language by first mastering basic units and then complex patterns.
- Potential for Low Bit-Rate Synthesis: The discovered AUs, even when reduced in number, show promise for achieving good synthesis quality with a lower bit-rate encoding.
Key Results
The proposed approach (System 1 and System 2) generally outperforms the baseline system across various evaluation metrics (MOS, CER, Similarity, ABX) on both English and Indonesian.
- Subjective Evaluation (MOS): System 1 achieved a MOS of 2.82 for English and 2.53 for Indonesian, outperforming the baseline (2.07 for Indonesian). The "top-line" system (a fully supervised ASR system) achieved higher MOS scores, especially for Indonesian (3.92), highlighting the advantage of transcribed data.
- Objective Evaluation (CER, Similarity, ABX): System 1 showed improvements in CER (0.55 vs. 0.62 for baseline in English) and Similarity (2.76 vs. 2.97 for baseline in English). ABX scores were also competitive.
- Bitrate: System 1 achieved a lower bitrate (138.59 kbps) compared to the baseline (71.98 kbps) for English, but the proposed system 2 (40 units) had a higher bitrate than the baseline despite fewer units. This is attributed to modeling each sonorant segment with three units (rising transient, steady state, falling transient).
- Indonesian Performance: While the proposed systems performed well, the top-line system significantly outperformed them on Indonesian, likely due to the near-perfect grapheme-phoneme correspondence in the language, which benefits supervised systems.
Limitations and Future Work
- Top-line Performance Gap: The proposed systems did not reach the performance of the fully supervised "top-line" system, particularly for Indonesian. This is attributed to the inherent advantage of having transcribed data.
- Bitrate Trade-off: While aiming for low bit-rate, System 2's bitrate was higher than the baseline, suggesting a trade-off between the number of units and encoding efficiency. The paper notes that modeling segments with three units contributes to this.
- Allophonic Variations: The paper acknowledges that allophonic variations in English can lead to inconsistencies, impacting MOS scores compared to supervised systems.
- Future Work: The authors suggest that the discovered units can be valuable for other zero-resource tasks like spoken term detection, where System 2 outperformed System 1. Further exploration of unit merging and modeling strategies could lead to improved synthesis quality and efficiency.
Summary updated