Improvement on plural unit selection and fusion

Jian Luan, Jian Li

Introduction

Concatenative speech synthesis generates high-quality audio by stitching together pre-recorded speech snippets, known as units. However, the quality of the resulting speech often suffers from "inter-unit" inconsistencies. Because snippets are extracted from different original sentences with varying prosody and emotional contexts, the transitions between them can sound unnatural or "unstable."

To mitigate this, researchers previously developed "plural unit selection," which selects multiple candidate units for a single target and fuses them to average out inconsistencies. While promising, early implementations relied on simple linear time-domain averaging, which frequently caused spectral blurring and quality degradation, particularly in complex sounds like diphthongs. This paper proposes an enhanced fusion pipeline using dynamic programming (DP) and spectral-domain processing to significantly improve naturalness.

Methodology

The proposed system introduces three critical improvements to the unit fusion process:

1. Dynamic Pitch Frame Alignment

Rather than using linear mapping to fit units to a target duration, the system employs Dynamic Programming (DP). By treating one unit as a template and aligning others to it based on maximal spectral similarity (within a Sakoe-Chiba search band), the system ensures that phonetic transitions—such as the start of a vowel—align correctly across all candidates. This is essential for Mandarin speech, which is rich in rapidly changing sounds like diphthongs and triphthongs.

2. Primary Unit Determination and Spectral Fusion

To maintain phase consistency and avoid the "mushy" sound of averaged phases, the system identifies a "primary unit" that is most similar to the other candidates. This unit provides the phase information for the final reconstruction.

The proposed spectral fusion pipeline, illustrating the transition from input frames through FFT-based magnitude and phase processing to the final output frame.

As shown in the pipeline above, the fusion occurs in the frequency domain:

  • Magnitude Fusion: Log-magnitude spectra are averaged across units. This acts as a filter, suppressing frequency components that are inconsistent across the candidate set.
  • Phase Preservation: The system adopts the phase spectrum of the primary unit entirely.
  • Reconstruction: The fused magnitude and primary phase are converted back to the time domain via Inverse Fast Fourier Transform (IFFT).

Experimental Results

The method was evaluated using two internal Mandarin speech corpora (one male, one female), each containing 3,000 sentences. A native-listener preference test was conducted using 25 generated sentences per narrator.

Preference test results comparing the proposed method against the baseline for male, female, and overall categories.

Listeners showed a strong preference for the proposed method. Overall, the new approach was preferred over the baseline by a ratio of 54% to 17%, with 29% of listeners showing no preference. Qualitative feedback indicated that the speech sounded "gentler" and that the most noticeable improvements occurred during complex vowels, confirming that DP alignment successfully reduced spectral mismatch.

Conclusion

By moving from time-domain averaging to DP-based spectral fusion, the authors significantly reduced artifacts in concatenative synthesis. While the method increases computational complexity due to the FFT/IFFT cycles and DP alignment, these costs can be managed by performing the fusion offline during database creation. This allows the real-time synthesis engine to benefit from higher-quality fused units without incurring additional latency during playback.

Summary updated

Abstract

Plural unit selection and fusion is a successful method for concatenative synthesis. Yet its unit fusion algorithm is simple and requires improvement. Previous research on unit fusion is mainly involved in boundary smoothing and not quite suitable for the application mentioned above. Therefore, a high-quality unit fusion method is proposed in this paper. More accurate pitch frame alignment and primary unit selection are implemented. Besides, the fusion of pitch frames is performed on FFT spectra for less quality loss. Experiment results indicate that the proposed method evidently outperforms the baseline at an overall preference ratio of 54:17.
BibTeX
@inproceedings{luan2010improvement,
  author = {Jian Luan and Jian Li},
  title = {{Improvement on plural unit selection and fusion}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-82},
}
Figure