Introduction
Concatenative speech synthesis generates high-quality audio by stitching together pre-recorded speech snippets, known as units. However, the quality of the resulting speech often suffers from "inter-unit" inconsistencies. Because snippets are extracted from different original sentences with varying prosody and emotional contexts, the transitions between them can sound unnatural or "unstable."
To mitigate this, researchers previously developed "plural unit selection," which selects multiple candidate units for a single target and fuses them to average out inconsistencies. While promising, early implementations relied on simple linear time-domain averaging, which frequently caused spectral blurring and quality degradation, particularly in complex sounds like diphthongs. This paper proposes an enhanced fusion pipeline using dynamic programming (DP) and spectral-domain processing to significantly improve naturalness.
Methodology
The proposed system introduces three critical improvements to the unit fusion process:
1. Dynamic Pitch Frame Alignment
Rather than using linear mapping to fit units to a target duration, the system employs Dynamic Programming (DP). By treating one unit as a template and aligning others to it based on maximal spectral similarity (within a Sakoe-Chiba search band), the system ensures that phonetic transitions—such as the start of a vowel—align correctly across all candidates. This is essential for Mandarin speech, which is rich in rapidly changing sounds like diphthongs and triphthongs.
2. Primary Unit Determination and Spectral Fusion
To maintain phase consistency and avoid the "mushy" sound of averaged phases, the system identifies a "primary unit" that is most similar to the other candidates. This unit provides the phase information for the final reconstruction.
As shown in the pipeline above, the fusion occurs in the frequency domain:
- Magnitude Fusion: Log-magnitude spectra are averaged across units. This acts as a filter, suppressing frequency components that are inconsistent across the candidate set.
- Phase Preservation: The system adopts the phase spectrum of the primary unit entirely.
- Reconstruction: The fused magnitude and primary phase are converted back to the time domain via Inverse Fast Fourier Transform (IFFT).
Experimental Results
The method was evaluated using two internal Mandarin speech corpora (one male, one female), each containing 3,000 sentences. A native-listener preference test was conducted using 25 generated sentences per narrator.
Listeners showed a strong preference for the proposed method. Overall, the new approach was preferred over the baseline by a ratio of 54% to 17%, with 29% of listeners showing no preference. Qualitative feedback indicated that the speech sounded "gentler" and that the most noticeable improvements occurred during complex vowels, confirming that DP alignment successfully reduced spectral mismatch.
Conclusion
By moving from time-domain averaging to DP-based spectral fusion, the authors significantly reduced artifacts in concatenative synthesis. While the method increases computational complexity due to the FFT/IFFT cycles and DP alignment, these costs can be managed by performing the fusion offline during database creation. This allows the real-time synthesis engine to benefit from higher-quality fused units without incurring additional latency during playback.
Summary updated