TL;DR
This paper introduces a novel statistical approach to restore high-frequency components in low-sampling-rate speech, crucial for improving the quality of synthesized speech in HMM-based systems. By integrating feature extraction and acoustic modeling within a unified generative framework, the method effectively leverages existing low-quality speech data for high-quality synthesis.
What Problem Does This Paper Address?
The quality of synthesized speech in HMM-based Text-to-Speech (TTS) systems is heavily reliant on the quality of training data. A significant challenge arises from the prevalence of older speech datasets recorded at lower sampling rates (e.g., 16 kHz) compared to modern high-sampling-rate recordings (e.g., 48 kHz). Directly using low-sampling-rate data degrades the synthesized speech quality, particularly in the spectral envelope. While re-recording or relabeling speech data is costly, this paper aims to bridge this gap by proposing a method to restore missing high-frequency spectral information from low-sampling-rate speech, thereby enabling more effective utilization of existing speech corpora.
Methods
The core of the proposed method lies in an "integration technique" that unifies feature extraction and acoustic modeling. Instead of treating these as separate steps, they are optimized jointly within a single generative model.
- Integrated Modeling of Speech Waveforms: High-sampling-rate speech waveforms are modeled directly using Gaussian Mixture Models (GMMs). This process optimizes the feature extraction and modeling stages simultaneously based on an integrated objective function. This is achieved by maximizing a likelihood function that considers both the speech waveform and the mel-cepstral coefficients. The optimization involves an Expectation-Maximization (EM) algorithm to iteratively refine model parameters.
- Restoring High-Frequency Components: Once the integrated model is trained on high-sampling-rate data, it's used as a prior distribution to estimate mel-cepstral coefficients from low-sampling-rate speech. The method aims to estimate mel-cepstral coefficients that restore the missing high-frequency spectral information. This is framed as maximizing the posterior probability of the mel-cepstral coefficients given the low-sampling-rate speech and the trained model. The probability of the mel-cepstral coefficients from the trained model acts as a prior, guiding the restoration of high-frequency components that are not present in the low-sampling-rate input.
- Avoiding Local Maxima: To address the sensitivity of the estimation to initial values, a deterministic annealing EM (DAEM) approach is employed. This involves modifying the objective function with a parameter $\beta$ that controls the balance between fitting the low-sampling-rate data and adhering to the prior model, gradually annealing $\beta$ to favor the data fit.
Datasets
- Training Data: High-sampling-rate speech waveforms were used to train the integrated GMM models. The paper mentions using 200 sentences for training models in the analysis-synthesis experiments and 250 sentences for training HMMs in the HMM-based synthesis experiments.
- Evaluation Data: The ATR Japanese speech database (Set B) [23], containing 503 phonetically balanced sentences uttered by a male speaker, was used. For evaluation, speech was downsampled from 48 kHz to 16 kHz.
The paper does not introduce new benchmarks but evaluates its method against existing techniques using a standard database.
Key Insights and Ideas
- Integration is Key: The paper strongly advocates for integrating feature extraction and acoustic modeling. This unified approach allows for optimization based on a single objective function, leading to more coherent and effective models compared to sequential processing.
- GMMs as Priors for Restoration: Using GMMs trained on high-quality data as prior distributions for estimating mel-cepstral coefficients from low-quality data is a powerful concept. It allows the model to "imagine" or reconstruct the missing high-frequency information based on learned patterns.
- Statistical Restoration: The method is fundamentally statistical, leveraging probabilistic models (GMMs) to infer and restore missing spectral information, rather than relying on deterministic signal processing techniques.
- Leveraging Existing Data: The primary impact is enabling the effective use of vast amounts of older, low-sampling-rate speech data for modern, high-quality TTS systems, saving significant costs associated with data acquisition.
Key Results
Subjective listening tests (Mean Opinion Score - MOS) were conducted for both analysis-synthesis and HMM-based speech synthesis.
- Analysis-Synthesis: The proposed method achieved MOS scores comparable to the original 48 kHz speech and significantly outperformed a conventional voice conversion (VC)-based method that converts mel-cepstra from 16 kHz to 48 kHz.
- HMM-based Speech Synthesis: Similar to the analysis-synthesis results, the HMM-based synthesis using mel-cepstral coefficients from the proposed method also showed a significant improvement in naturalness compared to the conventional method, with scores approaching those of the original 48 kHz speech.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the posterior distribution $Q(c_t|h_t)$ is estimated frame-by-frame for computational simplicity, which might not fully capture temporal dependencies.
- The method is currently demonstrated for a single speaker.
- Future Work:
- Objective evaluations are planned.
- Experiments with speaker-independent models are suggested to broaden the applicability of the technique.
Summary updated