TL;DR
This paper proposes a novel speech synthesis method that models the time-varying harmonic structure of phonemes using "Multiple Functions," specifically Multi-Gaussian Mixture Models (MGMM) and Multi-Beta Mixture Models (MBMM). This approach offers a more parameter-efficient way to synthesize speech compared to traditional methods, achieving comparable or better naturalness with significantly fewer parameters.
What Problem Does This Paper Address?
Traditional speech synthesis methods like concatenative synthesis, while producing natural speech, require massive datasets and computational resources. Additive synthesis, which models speech as a sum of sinusoids, is more efficient but often results in less natural-sounding speech. The challenge lies in finding a synthesis method that balances naturalness with computational efficiency and parameter reduction. This paper addresses this by focusing on modeling the harmonic-temporal structure of phoneme spectra, aiming to capture the essential characteristics of speech sound envelopes more efficiently.
Methods
The core innovation is the "Multiple Function" approach. Instead of directly modeling the entire spectrum, the authors extract harmonic components and model their time-varying intensity envelopes using a mixture model.
- Multiple Function: A general framework where the time-varying spectrum of each harmonic is modeled by a sum of "Partial Functions." The parameters of this function are estimated using an Expectation-Maximization (EM) algorithm.
- Partial Function: The building blocks of the Multiple Function. The paper explores two types:
- Multi-Gaussian Mixture Model (MGMM): Uses Gaussian Mixture Models (GMMs) as Partial Functions.
- Multi-Beta Mixture Model (MBMM): Uses Beta Mixture Models (BMMs) as Partial Functions. Beta distributions are chosen for their flexibility in modeling various shapes, particularly those with higher curvature, which can better represent spectral envelopes.
- Parameter Estimation: The paper details the derivation of update rules for MGMM and MBMM parameters using the EM algorithm and Kullback-Leibler (KL) divergence as an evaluation function.
- Speech Synthesis: Once the Multiple Function parameters are estimated for each phoneme, speech is synthesized using an additive synthesis approach, where the amplitude of each harmonic is determined by the learned Multiple Function.
Datasets
The experiments were conducted using recordings of five long vowels (/a:/, /e:/, /i:/, /o:/, /u:/) from a female speaker, sampled at 22.05 kHz. The recordings were pre-processed using Voice Activity Detection (VAD) and the PSOLA method to normalize pitch to 261 Hz. No new benchmark datasets were introduced; existing recordings were used for evaluation.
Key Insights and Ideas
- Harmonic-Temporal Structure is Key: The paper demonstrates that effectively modeling the time-varying intensity of harmonic components is crucial for capturing the essence of phoneme spectra.
- MBMM Superiority: The Multi-Beta Mixture Model (MBMM) generally outperformed the Multi-Gaussian Mixture Model (MGMM) in fitting the observed spectral envelopes. This is attributed to the Beta distribution's ability to model shapes with higher curvature, which better represent the dynamic spectral changes in speech.
- Parameter Efficiency: The proposed methods (MGMM and MBMM) require significantly fewer parameters than traditional additive synthesis methods while achieving comparable or better naturalness. This is a major advantage for storage and computational efficiency.
- Ignoring Noise: The improved naturalness observed with MGMM and MBMM compared to a baseline additive synthesis suggests that these models can effectively smooth out or ignore minor noise and fluctuations present in recorded speech, leading to a cleaner synthesized output.
Key Results
- Spectral Fitting: Both MGMM and MBMM models were found to represent the harmonic-temporal structure of phoneme signals well, capturing intensity ratios, rise/decay characteristics, and temporal aspects of spectral peaks.
- DP Distance: The Dynamic Programming (DP) distance metric showed that models with more mixtures (e.g., G3, B2) provided a closer fit to the original spectral structure. MBMM generally showed better performance than MGMM for the same number of mixtures.
- Mean Opinion Score (MOS): A listening test revealed that speech synthesized using MGMM (G3) and MBMM (B2) was perceived as more acoustically natural than the baseline additive synthesis method.
- Parameter Count: For example, with 20 harmonics and 4 mixtures, MBMM (B2) required only 260 parameters, whereas a baseline additive synthesis method (implicitly, by needing to store all harmonic parameters) required 21,240 parameters.
Limitations and Future Work
- Limitations:
- The paper focuses on synthesizing phoneme signals and does not cover the text analysis or prosody modeling required for full text-to-speech (TTS) systems.
- The current MBMM parameter estimation speed might be a bottleneck, although the paper mentions plans to improve it.
- Non-harmonic partials in voiced speech are not modeled.
- Future Work:
- Improve the iteration speed of MBMM parameter estimation.
- Devise more effective representing models.
- Integrate pitch contour and other phoneme modeling for more general speech synthesis.
- Extend the approach to model non-harmonic components.
Summary updated