TL;DR
This paper introduces an improved LPCNet vocoder, called iLPCNet, that uses a novel Linear Prediction-structured Mixture Density Network (LP-MDN). This approach enhances speech synthesis quality by better modeling the complex interactions between the vocal tract and vocal source, moving beyond the limitations of previous discrete representations and achieving a MOS score of 4.41.
What Problem Does This Paper Address?
Traditional text-to-speech (TTS) systems often rely on neural vocoders to generate realistic speech waveforms. While recent models like LPCNet have achieved high quality and efficiency by combining a linear prediction (LP) filter for spectral structure with a WaveRNN for vocal source generation, they face challenges. Specifically, the quality can be unstable because the vocal source (excitation) is often represented using a discretized format (µ-law quantization), which doesn't fully capture its nuances. Furthermore, these models don't explicitly model the entire speech production mechanism, leading to difficulties in representing the intricate interplay between the vocal tract and the excitation. This paper aims to address these limitations by proposing a more integrated and continuous approach to modeling speech production.
Methods
The core innovation is the Linear Prediction-structured Mixture Density Network (LP-MDN). The authors observe that the difference between the probability distributions of the speech waveform and its excitation signal can be simplified to a constant factor (the prediction term, $p_n$) when considering the LP synthesis process. By modeling both speech and excitation distributions using Mixture of Gaussians (MoG), they can derive the speech distribution parameters directly from the excitation distribution parameters and the prediction term.
The proposed iLPCNet integrates this LP-MDN into the existing LPCNet framework. Instead of the conventional LPCNet's discrete output, the iLPCNet uses the LP-MDN to generate continuous density distributions for the speech waveform. This allows the neural vocoder to directly learn the relationship between the LP filter and the excitation, effectively merging the speech production process into a single, end-to-end trainable model.
To further improve training, they introduce:
- STFT-based power loss: An auxiliary loss function that helps capture time-frequency characteristics of realistic speech, complementing the standard NLL loss.
- Conditional distribution sharpening: A technique to control the noisiness of generated speech by adjusting the scale parameters of the MoG distribution, applied selectively to voiced regions.
- Training noise injection: A method to inject Gaussian noise directly into the continuous linear domain, avoiding the computational overhead of µ-law conversion present in previous models.
Datasets
The experiments were conducted using a Korean female speaker corpus, sampled at 24 kHz with 16-bit quantization.
- Training: 4,976 utterances (9.9 hours)
- Validation: 280 utterances
- Testing: 140 utterances
The acoustic features used were 79-dimensional, extracted using an improved time-frequency trajectory excitation vocoder. For the text-to-speech evaluation, a Tacotron 2 model was employed as the acoustic model.
Key Insights and Ideas
- Bridging the discrete-continuous gap: The move from µ-law quantization to continuous density distributions (MoG) for speech generation is crucial for improving quality and stability. This allows for a more nuanced representation of the excitation signal.
- Structural integration of LP synthesis: By embedding the LP synthesis process directly into the neural network's structure (via LP-MDN), the model can learn the complex interactions between the vocal tract and vocal source more effectively. This is a departure from models that treat these components separately.
- Unified training and generation: The LP-MDN framework enables the model to learn and generate speech within the LP synthesis structure, simplifying the overall pipeline and potentially improving efficiency.
- Auxiliary losses matter: The STFT-based power loss demonstrates the benefit of incorporating perceptual audio quality metrics into the training objective, leading to more efficient and effective learning.
Key Results
- MOS Scores: The proposed iLPCNet achieved a Mean Opinion Score (MOS) of 4.41 in the text-to-speech (TTS) framework, a significant improvement over the conventional LPCNet (4.00) and WaveNet (3.33).
- Preference Test: Subjective preference tests showed that listeners significantly preferred the iLPCNet over the conventional LPCNet, especially in the TTS scenario.
- Model Complexity: The iLPCNet maintains a similar model complexity and generation speed to the conventional LPCNet, indicating an improvement in quality without a significant performance penalty.
- Simplified Pipeline: The removal of µ-law-specific submodules simplifies the LPCNet pipeline.
Limitations and Future Work
- Voiced Region Sharpening: The distribution sharpening technique was found to be most effective when applied only to the voiced regions. The authors don't explicitly discuss limitations of this selective application, but it suggests potential for further exploration in unvoiced regions.
- Single Gaussian for Speech: In the experiments, the distribution of the speech signal was defined as a single Gaussian. While this simplified the model, future work could explore using a full MoG for speech to potentially capture even more complex distributions.
- Further exploration of auxiliary losses: The paper mentions the STFT-based power loss and distribution sharpening. Future work could investigate other perceptual loss functions or sharpening strategies.
- Broader Acoustic Feature Sets: While the paper uses a specific set of acoustic features, exploring the impact of different feature representations on the iLPCNet's performance could be a direction for future research.
Summary updated