Improving Sequence-to-sequence Voice Conversion by Adding Text-supervision

Jing-Xuan Zhang, Zhen-Hua Ling, Yuan Jiang, Lijuan Liu, Liang Chen, Li-Rong Dai

Abstract

This paper presents methods for leveraging text supervision to enhance the performance of sequence-to-sequence (seq2seq) voice conversion (VC). While seq2seq acoustic modeling offers superior naturalness and similarity compared to traditional frame-to-frame approaches, it often suffers from stability issues such as mispronunciations and phoneme skipping. We propose two primary enhancements: a multi-task learning (MTL) framework that predicts linguistic labels via auxiliary classifiers, and a data-augmentation method utilizing text alignment to generate additional parallel training fragments. Experimental results demonstrate that these methods significantly reduce pronunciation errors and improve objective speech metrics, particularly in low-resource scenarios with as few as 50 training utterances.

Introduction

Voice Conversion (VC) technology aims to transform the speaker identity of a source utterance while preserving its linguistic content. Traditional frame-to-frame mapping often results in robotic speech due to its inability to model prosodic variations effectively. Seq2seq models address this by learning to adjust duration and rhythm. However, the reliance on attention mechanisms can lead to "stability" problems, where the model loses its place in the sequence, causing stutters, skips, or garbled words. This research explores how text transcriptions, which are typically available for parallel training data, can be used as a supervisory signal to regularize the model and ensure linguistic consistency.

Proposed Methodology

Multi-task Learning with Linguistic Labels

To stabilize the attention mechanism, we introduce auxiliary classifiers (AC) into the seq2seq architecture. These classifiers are attached to the encoder's output and the decoder's input, tasked with predicting phonemes and tones (linguistic labels) as a secondary objective. This forces the network's hidden layers to maintain a robust linguistic representation, acting as a constraint that prevents the model from deviating from the intended text during conversion.

Proposed multi-task learning architecture with auxiliary classifiers for linguistic labels.

Data Augmentation via Text Alignment

Seq2seq models are notoriously data-hungry. To improve performance in low-resource settings, we propose a data-augmentation technique based on silence-based slicing. By using text-to-speech alignment to identify pauses, we can segment long parallel utterances into multiple shorter, high-quality fragments. This process effectively multiplies the training samples without introducing the co-articulation artifacts that would occur if slicing were performed mid-word.

Experimental Evaluation

Setup

The system was evaluated using a Mandarin Chinese dataset consisting of 1,060 parallel utterances from one male and one female speaker. We tested the model across various training set sizes (50, 100, 200, 400, and 1,000 utterances) to assess its robustness in low-resource conditions. Features included 80-dimensional mel-spectrograms and 512-dimensional bottleneck features from a pre-trained speech recognition model.

Results and Discussion

Objective evaluations using Mel-cepstrum distortion (MCD) and F0 Root Mean Square Error (RMSE) show that the proposed text-supervised models consistently outperform the baseline seq2seq system. The multi-task learning approach (Seq2seq-MT) provides a baseline improvement in stability, while the data-augmentation (Seq2seq-MT-DA) offers substantial gains when training data is extremely limited (50–100 utterances).

Objective evaluation results (MCD and F0 RMSE) for different training data sizes across male-to-female and female-to-male tasks.

Subjective listening tests further confirmed these findings. Pronunciation errors dropped significantly; for instance, in the male-to-female task with 1,000 utterances, the number of mistakes fell from 12 in the baseline to just 3 in the proposed model. ABX preference tests indicated that listeners overwhelmingly preferred the naturalness and speaker similarity of the text-supervised models, especially in the 50-utterance scenario.

Conclusion

By integrating text supervision through multi-task learning and data augmentation, we have significantly improved the stability and quality of seq2seq voice conversion. These techniques are particularly effective for low-resource speaker adaptation. Future work will focus on applying these methods to even smaller datasets and exploring cross-lingual voice conversion applications.

Summary updated

Abstract

This paper presents methods of making using of text supervision to improve the performance of sequence-to-sequence (seq2seq) voice conversion. Compared with conventional frame-to-frame voice conversion approaches, the seq2seq acoustic modeling method proposed in our previous work achieved higher naturalness and similarity. In this paper, we further improve its performance by utilizing the text transcriptions of parallel training data. First, a multi-task learning structure is designed which adds auxiliary classifiers to the middle layers of the seq2seq model and predicts linguistic labels as a secondary task. Second, a data-augmentation method is proposed which utilizes text alignment to produce extra parallel sequences for model training. Experiments are conducted to evaluate our proposed method with training sets at different sizes. Experimental results show that the multi-task learning with linguistic labels is effective at reducing the errors of seq2seq voice conversion. The data-augmentation method can further improve the performance of seq2seq voice conversion when only 50 or 100 training utterances are available.
BibTeX
@inproceedings{zhang2019improving,
  author = {Jing-Xuan Zhang and Zhen-Hua Ling and Yuan Jiang and Lijuan Liu and Liang Chen and Li-Rong Dai},
  title = {{Improving Sequence-to-sequence Voice Conversion by Adding Text-supervision}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=w4sHr23z7a},
}
Figure