TL;DR
This paper introduces a simple yet effective method to improve the naturalness of speech synthesis by reducing prominent pitch peaks in the original recordings. By applying this technique to the training data, the authors demonstrate a significant improvement in the perceived quality of both concatenative and HMM-based synthesis systems, without negatively impacting intelligibility.
What Problem Does This Paper Address?
Creating high-quality speech synthesis systems, especially those based on existing speech corpora (like concatenative or HMM-based systems), heavily relies on the quality of the source recordings. A common issue in professional speech recordings is the presence of pronounced pitch peaks. These peaks, often used as a stylistic device, can lead to noticeable and jarring discontinuities in the synthesized speech, negatively affecting its naturalness. While discarding problematic recordings is an option, it's costly and time-consuming. This paper addresses the challenge of improving synthesized speech quality by proposing a method to correct these pitch peak issues directly within the existing corpus, rather than resorting to expensive re-recordings.
Methods
The core of the paper's contribution lies in a method to attenuate pronounced pitch peaks in the source audio recordings. This is achieved by manipulating the pitch track using the Time-Domain Pitch-Synchronous OverLap-and-Add (TD-PSOLA) technique. Specifically, they employ a formula that linearly scales the pitch track by a factor s above a certain threshold T, leaving it unchanged below T. After experimenting with various parameters, they settled on a threshold T = 200Hz and a scaling factor s = 0.4. This means that pitch values above 200Hz are reduced by 60%, while those below remain untouched. This modified pitch track is then used to reconstruct the audio.
To evaluate this method, the authors built eight speech synthesizers by combining two frameworks (Festival and MARY), two synthesis approaches (concatenative and HMM-based), and two versions of the training corpus (original and modified with pitch peak reduction).
Datasets
The primary dataset used in this work is the SECYT corpus. This corpus consists of 741 declarative sentences recorded by a female professional speaker of Argentine Spanish. It is characterized by a pitch range of 130-380Hz, an average sentence length of 7 words and 3.9 seconds, and includes manual phonetic transcriptions and time alignments.
For intelligibility evaluation, they used 50 Semantically Unpredictable Sentences (SUS), each 6-10 words long, created by Gurlekian et al. (2012). For naturalness evaluation, they used a set of 20 sentences (5-20 words long) from Gurlekian et al. (2012) and an additional 20 sentences created for this study.
Key Insights and Ideas
- Pitch peaks are a significant source of unnaturalness: The paper strongly suggests that pronounced pitch peaks, even in professional recordings, are a major culprit behind discontinuities and reduced naturalness in corpus-based speech synthesis.
- Targeted pitch modification is effective: The TD-PSOLA based pitch scaling method, while simple, proves to be a powerful tool for improving the quality of the source recordings without introducing new artifacts.
- Naturalness gains without intelligibility loss: A crucial finding is that reducing pitch peaks improves perceived naturalness without any detrimental effect on the intelligibility of the synthesized speech. This is a highly desirable outcome for speech synthesis development.
- Framework and synthesis type matter: The effectiveness of the pitch reduction method varies across different synthesis systems. MARY-based systems showed significant improvements, while Festival-based systems showed mixed results, with the concatenative system showing no improvement. This highlights the interplay between the modification technique and the underlying synthesis architecture.
Key Results
- Intelligibility: The intelligibility of all eight synthesized systems (across both frameworks and synthesis types) was not significantly affected by the pitch peak reduction method. Wilcoxon signed-rank tests showed no significant differences between the original and modified corpus versions for any of the systems.
- Naturalness:
- MARY-based systems: Both the concatenative (
mary_conc) and HMM-based (mary_hmm) systems built using the modified corpus showed significantly higher perceived naturalness.
- MARY-based systems: Both the concatenative (
- Festival-based systems: The HMM-based system (
fest_hmm) showed an improved naturalness that approached statistical significance (p=0.126). The concatenative system (fest_conc) showed no significant improvement in naturalness. - Overall: The proposed method did not harm intelligibility and managed to improve naturalness for most of the tested systems, particularly those built with the MARY framework.
Limitations and Future Work
- Framework dependency: The paper notes that the improvements in naturalness were not uniform across all systems, with Festival's concatenative system showing no gains. This suggests that the effectiveness of the method might depend on the specific implementation details of the synthesis framework and how it handles pitch variations.
- Limited scope of modification: The method focuses solely on reducing pitch peaks. Other potential issues in speech corpora, such as intensity variations or subtle acoustic artifacts, are not addressed.
- Future Work: The authors suggest that developers might find this methodology beneficial for improving corpus-based speech synthesizers. While not explicitly detailed, future work could explore applying this method to a wider range of speech corpora and synthesis systems, or investigate more complex pitch modification strategies.
Summary updated