TL;DR
This paper introduces a novel approach to training unit selection speech synthesis systems by learning a "target cost" function directly from human perceptual judgments. Instead of relying on complex, hand-tuned acoustic features, they train a classifier to predict whether a synthesized speech unit is "good" or "bad." This method achieves performance nearly as good as traditional, expertly tuned systems but with significantly less manual effort.
What Problem Does This Paper Address?
Unit selection speech synthesis systems select and concatenate pre-recorded speech units (like diphones) to generate new speech. A crucial component is the "target cost" function, which guides the selection of the best units based on linguistic and acoustic features. Traditionally, this target cost is a complex, weighted sum of various factors that requires significant human expertise and time to tune. This manual tuning is problematic because:
- Subjectivity: What sounds "good" is subjective and can vary. Traditional methods often rely on acoustic similarity to natural speech, which doesn't fully capture perceptual acceptability.
- Complexity: The interactions between different acoustic and linguistic features are complex and unlikely to be linear, making hand-tuning difficult.
- Effort: Developing and tuning these functions is time-consuming and requires specialized knowledge.
The authors aim to automate the learning of this target cost function by directly leveraging human perception, making the process more efficient and potentially more accurate by focusing on what listeners actually hear.
Methods
The core innovation is training a classifier to predict perceptual quality rather than using a hand-crafted, continuous-valued cost function.
- Perceptual Data Collection:
- They synthesized a large number of short sentences (145,137) with the standard target cost turned off, so the selection was driven by join costs.
- They selected 7,200 sentences with the "smoothest" joins.
- 60 listeners were asked to mark "bad" syllables within these sentences.
- This syllable-level rating was converted to a binary "good" or "bad" label for each demiphone (half-phone unit used in synthesis).
- Feature Engineering:
- Input features for the classifier included linguistic context features for the target phone, as well as features describing the left and right candidate demiphones.
- Categorical features were converted to numerical representations.
- The final feature vector length was 151 for demiphone-level prediction.
- Classifier Training:
- They experimented with different classifiers: Classification Trees (CART), Timbl (k-NN), and Support Vector Machines (SVMs).
- SVMs with a polynomial kernel (order 2, parameter s=0.06) performed best.
- To handle the diphone-based nature of Festival's synthesis, they trained four separate SVMs: one for left/right demiphones of vowels and one for left/right demiphones of consonants.
- Integration into Festival:
- For integration, the SVM output (signed distance to the hyperplane) was transformed into a cost.
- They used a 2-class ("good"/"bad") and a 3-class ("good"/"fair"/"bad") system.
- Crucially, they identified specific context features ("boundary tone" or "emphasis" mismatch) that are always perceptually bad and assigned a very high cost (999) directly, bypassing the SVM for these cases. Otherwise, the SVM output was mapped to costs of 0 ("good") or 99 ("bad").
Datasets
- Training Data: 7,200 short sentences synthesized from the British National Corpus.
- Evaluation Data:
- 400 sentences from the Blizzard 2008 test set.
- 540 short sentences (2-4 words) similar to the training data.
No new benchmark datasets were introduced; they used existing test sets for evaluation.
Key Insights and Ideas
- Perception is Key: The most significant insight is that directly learning from human perceptual judgments is a viable and effective alternative to relying solely on acoustic similarity metrics.
- Classifier Simplicity: Using a classifier to make a binary (or ternary) decision about unit quality is sufficient and more robust than trying to estimate a fine-grained, continuous cost. This acknowledges that a single "bad" unit can ruin an entire utterance, making the primary goal to avoid bad units.
- Automation Advantage: This approach significantly reduces the manual effort and expertise required for tuning target cost functions, making it easier to build high-quality voices in new languages or for less experienced developers.
- Handling Critical Errors: Explicitly handling perceptually critical mismatches (like boundary tone) with a high fixed cost is an effective way to ensure basic quality, even when the classifier might not perfectly predict these extreme cases.
Key Results
- Performance: The classifier-based target cost achieved performance "almost as good as" the carefully and expertly tuned standard target cost.
- SVM Performance: SVMs outperformed CART and Timbl, achieving up to 71.5% accuracy on individual phones and 66.6% on all phones.
- Listening Test Results:
- In a listening test with 400 Blizzard sentences, listeners preferred the default target cost 53.4% of the time, meaning the new cost was preferred 46.6% of the time.
- A second test with 540 shorter sentences yielded similar results (54.3% preference for default, 45.7% for new cost).
- The difference in preference was statistically significant but small, indicating the new method is competitive.
- Data Efficiency: The authors suggest that their results are close to convergence even with a relatively modest amount of perceptual data (72 person-hours).
Limitations and Future Work
- Limitations:
- The paper acknowledges that the classifier-based cost is "almost as good" as the standard cost, implying a slight performance gap.
- The mapping of classifier output to continuous costs (0 and 99) is somewhat arbitrary.
- The "bad" label for a syllable is pessimistically applied to all its constituent phones/demiphones.
- Future Work:
- The authors suggest that a third "fair" class, defined by a range of signed distances from the SVM hyperplane, was preferred in a pilot test and could be further explored.
- The paper implies that further tuning of the classifier parameters or exploring different classifier architectures could yield further improvements.
- The approach's suitability for languages with less speech synthesis expertise is highlighted as a promising direction.
Summary updated