TL;DR
This paper introduces a novel "psychoacoustic segment scoring" method to intelligently decide whether to use pre-recorded speech segments or statistically generated ones in multi-form speech synthesis. This allows for higher quality speech even when a large portion of the output is synthesized, reducing audible artifacts at the transitions between recorded and synthesized speech.
What Problem Does This Paper Address?
Modern speech synthesis systems often combine pre-recorded speech segments (templates) with statistically generated segments (models) to achieve a balance between naturalness and flexibility. This is known as Multi-Form Segment (MFS) synthesis. A key challenge is determining the optimal "Model-Template Ratio" (MTR) – how much of the speech should be synthesized. While higher MTR offers more flexibility, it can lead to audible artifacts and reduced quality, especially at the boundaries where synthesized and recorded speech meet. This paper aims to improve MFS synthesis by developing a more intelligent way to select segment representations, allowing for higher MTRs without sacrificing speech quality.
Methods
The core innovation is a psychoacoustic segment scoring technique. This method assesses the "stationarity" of a speech segment – how much its acoustic properties change over time. The intuition is that more stationary segments are better suited for statistical modeling, while less stationary segments (like transients) are better preserved as pre-recorded templates.
Here's a breakdown of the key methods:
- Stationarity Score (S): The paper defines a score based on the perceptual loudness spectrum (PLS) of frames within a segment. It measures the variability of these loudness vectors to quantify stationarity.
- Leaf Stationarity Measure (LSM): This extends the segment stationarity to an "acoustic leaf" (a unit in the speech synthesis model). It uses the lower bound of stationarity scores from segments associated with that leaf.
- Leaf Loudness Measure (LLM): Similar to LSM, this measures the loudness of acoustic leaves.
- Leaf Modeling Suitability Factor (LMF): This is a combined score that weighs both stationarity (LSM) and loudness (LLM). Leaves with high LMF are deemed suitable for statistical modeling.
- Model-Template Ratio (MTR) Determination: Based on the LMF scores, leaves are ranked. A target MTR percentage then determines how many of the top-ranked leaves will be synthesized using statistical models, while the rest will be used as templates.
- Boundary-Constrained Generation: To further improve quality at high MTRs, the paper introduces techniques to constrain the generation of acoustic parameters for model segments. This ensures smoother transitions at model-template joints.
- Waveform Discontinuity Reduction: Methods are presented to minimize audible clicks or pops at segment joints. This includes phase alignment for voiced segments and a multi-peak cross-correlation analysis for template-template joints to find optimal concatenation points.
Datasets
Not explicitly detailed in the paper, but the authors mention building MFS voices from "mid-size voice datasets" and using "large scale subjective evaluations" involving both in-house participants and crowdsourced evaluators.
Key Insights and Ideas
- Stationarity as a Key Cue: The paper highlights that segment stationarity, measured psychoacoustically, is a powerful indicator for deciding between template and model representations, even in isolation from phonetic cues.
- Trade-off Between Stationarity and Loudness: The LMF score elegantly balances stationarity and loudness. While stationary segments are good for modeling, very silent stationary segments might not be ideal if they introduce audible artifacts when joined with louder template segments.
- High MTR is Achievable: The proposed methods demonstrate that it's possible to achieve high MTRs (up to 60% and even 100% in some evaluations) with minimal perceived quality degradation, a significant improvement over previous approaches.
- Crowdsourcing Reveals Different Perceptual Differences: The crowdsourcing evaluation suggested that listeners less sensitive to fine acoustic details (e.g., not using hi-fi headphones) might even prefer MFS speech at moderate MTRs over purely concatenative speech, likely due to the reduction of subtle discontinuities in the latter.
Key Results
- Subjective evaluations showed that MFS speech generated with the proposed psychoacoustic scoring and refinement techniques maintained high quality even at MTR levels of 25% and 40%.
- At MTR=60%, a relatively small perceived quality degradation was observed, and this speech was rated substantially higher than purely statistical parametric speech (MTR=100%).
- The crowdsourcing evaluation indicated that MTR=25% samples received the highest average rating, even outperforming the reference (MTR=0% or fully concatenative mode) in some instances.
Limitations and Future Work
- Limitations: The paper acknowledges that while the psychoacoustic score is used in isolation, it can be augmented by phonetic and acoustic cues for even better performance. The current method might lead to quality degradation if only the stationarity score is used without considering loudness.
- Future Work: The authors suggest further refinements to the concatenation techniques and exploring the interplay of their proposed method with other acoustic and phonetic cues for even more robust MFS synthesis.
Summary updated