TL;DR
This paper introduces a novel neural network implementation of Vocal Tract Length Normalization (VTLN) for Text-to-Speech (TTS) systems. By integrating VTLN as a trainable layer within a deep neural network, the authors enable automatic learning of speaker-specific spectral warping parameters, leading to improved TTS quality and more efficient speaker adaptation with less data.
What Problem Does This Paper Address?
Text-to-Speech (TTS) systems aim to generate human-like speech. A key challenge in TTS is achieving natural-sounding speech for different speakers, which often requires adapting a generic model to a specific voice. Vocal Tract Length Normalization (VTLN) is a well-established technique, originally from Automatic Speech Recognition (ASR), that can help normalize for speaker differences by accounting for variations in vocal tract length. VTLN essentially warps the spectral features of speech. While VTLN is known to be effective and requires less adaptation data than full model retraining, its integration into modern deep learning-based TTS systems has been limited. Traditional methods often involve separate VTLN parameter estimation steps, which can be cumbersome and not fully integrated into the end-to-end learning process. This paper addresses the gap of seamlessly incorporating VTLN into deep neural network TTS architectures for efficient and effective speaker adaptation.
Methods
The core innovation of this paper is the development of a "Neural VTLN" layer that can be integrated directly into a deep neural network. This layer takes mel-cepstral coefficients (a common representation of speech features) as input and, crucially, predicts the VTLN warping parameter ($\alpha$) on a frame-by-frame basis. This predicted $\alpha$ is then used to compute a warping matrix, which transforms the input mel-cepstral coefficients.
The authors propose a method to compute this warping matrix efficiently within a neural network framework. They leverage a mathematical formulation of VTLN that expresses the warping matrix as a function of $\alpha$. By pre-computing a 3D tensor containing polynomial coefficients and then performing a dot product with the predicted $\alpha$ vector, they can generate the exact warping matrix for each frame. This approach allows the entire VTLN layer, including the prediction of $\alpha$ and the subsequent warping, to be trained end-to-end using backpropagation, just like any other layer in the neural network. This is a significant departure from previous methods that often relied on iterative or separate optimization steps for VTLN parameters.
Datasets
The primary dataset used in this work is the VCTK database. The authors specifically utilized recordings from 33 speakers with English accents. For experiments, they extracted phone sequences and performed force-alignment using HMMs. They also created an artificial speaker by warping the MGC features of a base speaker (speaker p276, female) with known, phoneme-dependent warping parameters. This artificial dataset was crucial for validating the network's ability to learn ground-truth warping parameters.
Key Insights and Ideas
- End-to-End VTLN Learning: The most significant insight is that VTLN parameters can be learned directly through backpropagation within a neural network. This eliminates the need for separate VTLN estimation steps and allows for a fully integrated TTS system.
- Efficient Neural VTLN Layer: The proposed method for computing the VTLN warping matrix within the neural network is computationally efficient and differentiable, enabling seamless integration into deep learning pipelines.
- Phoneme-Dependent Warping: The experiments with the artificial speaker demonstrate that the neural network can learn specific, phoneme-dependent warping parameters, suggesting a fine-grained understanding of speaker characteristics.
- Data Efficiency: The paper highlights that VTLN, due to its small parameter space, offers a data-efficient approach to speaker adaptation in TTS, outperforming baseline methods with limited adaptation data.
Key Results
- Artificial Speaker Adaptation: The neural VTLN layer learned to compensate for approximately 41% of the error introduced by artificial warping on an artificial speaker. This compensation was more effective in lower frequency bins, as expected for VTLN.
- Multi-Speaker TTS Performance: In a multi-speaker TTS system, the proposed VTLN model achieved better objective scores (lower MCD, Fo RMSE, and BAP) compared to a strong baseline system, even when trained from scratch.
- Speaker Adaptation Performance: For speaker adaptation with limited data (as few as 10 utterances per speaker), the VTLN system significantly outperformed the baseline in both objective and subjective listening tests. In a preference test, 50.8% of listeners preferred the VTLN system, compared to 23.2% for the baseline.
Limitations and Future Work
- Numerical Instability: The authors note that their implementation of the VTLN matrix computation can become numerically unstable for a large number of mel-cepstral coefficients, suggesting the use of double precision computation in such cases.
- Limited Comparison: The paper acknowledges that they are not conducting an exhaustive comparison with all state-of-the-art techniques in speaker adaptation, as their primary focus is on demonstrating the neural VTLN concept.
- Future Work: The authors plan to test the proposed neural VTLN technique in emotional speech synthesis, where formant shifts are known to be influenced by emotions. They also suggest that the approach is applicable to Encoder-Decoder TTS models.
Summary updated