TL;DR
This paper introduces a novel method for extracting spectral envelope features from speech using a Riesz transform-based demodulation technique. These Riesz-derived features, when used to condition a WaveNet vocoder, result in synthesized speech that is objectively and subjectively superior to that generated using features from the widely-used WORLD vocoder.
What Problem Does This Paper Address?
Synthesizing high-quality speech is a crucial task for many applications, from virtual assistants to translation services. A key component in speech synthesis is the vocoder, which models the characteristics of speech. Traditional vocoders rely on source-filter models, estimating parameters like pitch, spectral envelope, and aperiodicity. More recently, neural vocoders like WaveNet have shown great promise, learning the mapping from acoustic features to raw audio samples. However, the quality of synthesized speech from these neural vocoders is highly dependent on the choice of acoustic features used for conditioning. This paper addresses the problem of finding more effective acoustic features for spectral envelope representation, aiming to improve the quality of WaveNet-based speech synthesis.
Methods
The core of this paper lies in a novel spectro-temporal demodulation technique. Instead of relying on traditional spectrograms, the authors propose a "pitch-adaptive spectrogram" where the analysis window length is varied in proportion to the speech signal's fundamental period. This pitch-adaptive spectrogram is then modeled as a 2D amplitude- and frequency-modulated (AM-FM) cosine signal.
The key innovation is the use of the Riesz transform to demodulate this signal and extract the AM component, which represents the vocal-tract filter's magnitude response (the spectral envelope). This is a departure from traditional methods that might use Hilbert transforms or other signal processing techniques. The authors also introduce a formant bandwidth correction method using a weighted central difference operator to refine the extracted envelope.
These Riesz-derived spectral envelope features are then used as local conditioning for a WaveNet vocoder. For comparison, they also train a WaveNet vocoder using spectral envelope features extracted from the state-of-the-art WORLD vocoder.
Datasets
The primary dataset used for training and evaluation is the CMU-ARCTIC speech database. The authors used four different voices from this database: two male (BDL and RMS) and two female (SLT and CLB). The original sampling frequency of 32 kHz was downsampled to 16 kHz for simplification. For each speaker, 1000 sentences were used for training, 50 for validation, and the remaining for testing.
Key Insights and Ideas
- Riesz Transform for Spectral Envelope: The Riesz transform, a generalization of the Hilbert transform to higher dimensions, proves to be a powerful tool for accurately demodulating spectro-temporal signals and extracting the spectral envelope.
- Pitch-Adaptive Spectrogram Matters: Adapting the analysis window to the speech pitch helps in better capturing the signal's characteristics, providing a more suitable input for the demodulation process.
- AM-FM Modeling is Effective: Modeling spectro-temporal patches as AM-FM signals allows for a more nuanced representation of speech characteristics, with the AM component directly relating to the spectral envelope.
- Formant Bandwidth Correction is Crucial: Refining the extracted envelope by correcting formant bandwidths leads to more natural-sounding synthesized speech, as accurate formant bandwidths are known to be important for speech synthesis quality.
- Feature Quality Directly Impacts Synthesis Quality: The study strongly reinforces the idea that the choice of acoustic features is paramount for the performance of neural vocoders like WaveNet. Better features lead to demonstrably better synthesized speech.
Key Results
The paper presents both objective and subjective evaluations of the synthesized speech quality.
- Objective Measures: Using STOI (Short-Time Objective Intelligibility) and PESQ (Perceptual Evaluation of Speech Quality) scores, the WaveNet vocoder conditioned with the Riesz envelope consistently outperformed the WaveNet vocoder conditioned with the WORLD envelope across all tested speakers. For example, PESQ scores for the proposed method were generally higher (e.g., 2.86±0.25 for BDL) compared to WORLD (e.g., 2.69±0.41 for BDL). STOI scores were also consistently higher for the Riesz method.
- Subjective Measures (MOS): An informal listening test using Mean Opinion Scores (MOS) revealed that subjects rated the speech synthesized using the Riesz envelope significantly higher than that synthesized using the WORLD envelope. For instance, MOS scores for the proposed method were around 4.67-4.76, while WORLD scores were around 3.07-3.47, indicating a substantial improvement in perceived quality.
Limitations and Future Work
- Computational Cost: While not explicitly detailed as a limitation, the Riesz transform and demodulation process might be computationally more intensive than some traditional methods.
- Generalizability: The study focuses on a specific set of voices from the CMU-ARCTIC database. Further evaluation on a wider range of speakers, accents, and noisy conditions would be beneficial.
- FM Component: The paper primarily focuses on the AM component for the spectral envelope. The authors have previously shown the importance of the FM component for other speech tasks, and its potential role in improving synthesis quality could be explored further.
- Future Work: The authors suggest exploring the use of the FM component and further investigating formant bandwidth correction techniques. They also imply that this Riesz-based approach could be beneficial for other speech processing tasks beyond synthesis.
Summary updated