TL;DR
This paper delves into a probabilistic model for concatenating speech units, moving beyond simple distance metrics to predict how well units will join. By framing concatenation cost as a conditional Gaussian probability, the authors demonstrate improved objective evaluation scores and explore the model's flexibility, showing it can encompass other common methods as special cases.
What Problem Does This Paper Address?
Creating natural-sounding synthetic speech often relies on concatenating pre-recorded speech units (like phonemes or diphones). A critical challenge is ensuring these units blend seamlessly at their boundaries. Traditional methods often use "distance" metrics between acoustic features of adjacent units to estimate this "join cost." However, these distance-based approaches might not fully capture the perceptual quality of the join. This paper investigates a probabilistic approach that models the likelihood of a unit's spectral shape given the preceding unit's shape, aiming for a more perceptually relevant measure of concatenation quality.
Methods
The core of the paper is the Probabilistic Concatenation Model (PCM), which uses a conditional Gaussian probability density function (PDF) to model the "goodness" of a concatenation. Specifically, it models the probability of observing the spectral shape of the current unit's "head" ($h(u_i)$) given the spectral shape of the previous unit's "tail" ($t(u_{i-1})$). This relationship is expressed as:
$P(h(u_i) | t(u_{i-1}), S, i) = \mathcal{N}(h(u_i) | B t(u_{i-1}) + b, \Sigma)$
Here:
- $h(u_i)$ is the d-dimensional spectral vector of the current unit's head.
- $t(u_{i-1})$ is the d-dimensional spectral vector of the previous unit's tail.
- $B$ is a $d \times d$ regression matrix.
- $b$ is a d-dimensional intercept vector.
- $\Sigma$ is a $d \times d$ covariance matrix.
These parameters ($B, b, \Sigma$) are learned from training data using Maximum Likelihood (ML) estimation. The paper also details how to estimate these parameters efficiently by reformulating the problem into a linear regression framework. A key aspect is the flexibility of this model:
- If $B$ is close to the identity matrix and $b$ is close to zero, the model approximates a scenario where the head and tail spectra are very similar (akin to Euclidean distance).
- If $B$ is close to zero, the head spectrum is largely independent of the tail, and $b$ dominates the mean.
Datasets
- Training Data: The paper mentions using a corpus for developing a speech synthesizer, which includes prompt sentences and unit databases. For parameter estimation, pairs of tail and head spectral feature vectors from unit boundaries are used.
- Evaluation Data:
- Closed Test Set: A set of 29 prompt sentences from the same corpus used for training.
- Open Test Set: 50 conversational sentences ("conv") used in the Blizzard Challenge 2005.
Key Insights and Ideas
- Probabilistic View is Superior: The paper strongly suggests that modeling concatenation as a probabilistic event, rather than just a distance, leads to better objective evaluation results and aligns better with perceptual quality.
- Flexibility is Key: The conditional Gaussian model is not just a single approach but a framework. Its ability to represent various relationships between unit boundaries (from near-identity transformations to near-independence) makes it powerful.
- Relationship to Existing Methods: The paper mathematically demonstrates how standard Euclidean distance and Donovan's Mahalanobis distance can be seen as special cases of the proposed probabilistic model, highlighting its generality.
- Objective Evaluation Corroborates Subjective Findings: While a previous paper showed subjective improvements, this work provides objective evidence (MFCC distance via DTW) that the probabilistic model leads to synthetic speech closer to natural speech.
Key Results
- Subjective Listening Test (from prior work): The proposed method achieved significantly higher mean opinion scores than a baseline Euclidean distance method.
- Objective Evaluation (Closed Test Set): The proposed method achieved a smaller average MFCC distance (16.1 $\pm$ 2.26) compared to the Euclidean baseline (17.7 $\pm$ 0.98), a statistically significant improvement at the 1% level.
- Objective Evaluation (Open Test Set): The proposed method also showed improvement (18.31 $\pm$ 1.21) over the Euclidean baseline (18.48 $\pm$ 1.13), significant at the 5% level, though the gap was smaller than in the closed test.
- Analysis of Concatenation Scores: Plots of concatenation scores across an utterance show that the probabilistic model can produce smoother score sequences, especially when the "correct" unit sequence is used, suggesting it captures more meaningful phonetic context.
Limitations and Future Work
- Data Sparsity: The paper acknowledges that for some phone pairs, training data might be sparse. They address this by using phonetic decision-tree clustering to tie model parameters.
- Model Complexity: While flexible, estimating the parameters for a large number of phone pairs can be computationally intensive.
- Future Work: The paper implies that further exploration of the model's flexibility and its application to different phonetic contexts or feature types could be beneficial. The analysis of score sequences also suggests potential for more sophisticated unit selection strategies based on these probabilistic scores.
Summary updated