Abstract
This paper presents methods of making use of text supervision to improve the performance of sequence-to-sequence (seq2seq) voice conversion. Compared with conventional frame-to-frame voice conversion approaches, the seq2seq acoustic modeling method proposed in our previous work achieved higher naturalness and similarity. In this paper, we further improve its performance by utilizing the text transcriptions of parallel training data. First, a multi-task learning structure is designed which adds auxiliary classifiers to the middle layers of the seq2seq model and predicts linguistic labels as a secondary task. Second, a data-augmentation method is proposed which utilizes text alignment to produce extra parallel sequences for model training. Experiments are conducted to evaluate our proposed method with training sets at different sizes. Experimental results show that the multi-task learning with linguistic labels is effective at reducing the errors of seq2seq voice conversion. The data-augmentation method can further improve the performance of seq2seq voice conversion when only 50 or 100 training utterances are available.
Introduction
Sequence-to-sequence (seq2seq) models have revolutionized voice conversion (VC) by eliminating the need for strict frame-by-frame alignment between source and target speakers. However, these models often suffer from instability, leading to "stuttering," skipped syllables, or mispronunciations. These issues arise because the models learn purely from acoustic features, and the attention mechanism can easily lose track of the linguistic content. This paper addresses these challenges by re-introducing text transcriptions into the training process as a form of supervision to stabilize the conversion process, particularly in low-resource scenarios.
Proposed Methods
Multi-Task Learning (MTL) with Linguistic Labels
The core of the proposed improvement is a multi-task learning framework that forces the model to maintain a clear representation of the spoken content. As shown in the architecture below, auxiliary classifiers (AC) are integrated into the encoder's output and the decoder's input. During training, these classifiers are tasked with predicting linguistic labels, such as phonemes and tones, alongside the primary acoustic conversion task. This dual-objective approach acts as a regularizer, ensuring the hidden layers capture robust linguistic features that guide the attention mechanism.
Data Augmentation (DA) by Text Alignment
To improve performance when training data is scarce, the authors propose a data augmentation method based on text-to-speech alignment. By identifying silence points between words, long sentences are sliced into multiple smaller parallel fragments. This process effectively multiplies the training samples and forces the model to learn granular relationships between source and target voices, rather than over-fitting to specific long-sentence contexts.
Experimental Results
Objective Evaluation
The proposed methods were evaluated using a Mandarin parallel corpus with training sets ranging from 50 to 1000 utterances. Objective metrics, including Mel-cepstrum distortion (MCD) and Fundamental Frequency Root Mean Square Error ($F_0$ RMSE), demonstrate significant improvements. As shown in the table below, the combination of MTL and DA (Seq2seq-MT-DA) consistently outperforms the baseline, especially in low-resource settings (e.g., 50 or 100 utterances).
Subjective Evaluation and Stability
Subjective tests confirmed that text supervision dramatically reduces pronunciation errors. In the Male-to-Female task using 1000 training utterances, the number of utterances containing skips or repeats dropped from 12 in the baseline to just 3 with the MTL approach. Preference tests also indicated that listeners favored the proposed method for its superior naturalness and speaker similarity, particularly when training data was limited.
Conclusion
By incorporating text supervision through multi-task learning and data augmentation, the stability and quality of seq2seq voice conversion are significantly enhanced. While data augmentation is most beneficial in low-resource scenarios, multi-task learning with linguistic labels provides a consistent performance boost across all dataset sizes by grounding acoustic modeling in linguistic reality.
Reference: Zhang, J. X., Ling, Z. H., Jiang, Y., Liu, L. J., Liang, C., & Dai, L. R. (2019). Improving Sequence-to-Sequence Voice Conversion by Adding Text-Supervision. arXiv:1811.08111.
Summary updated