Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision

Jing-Xuan Zhang, Zhen-Hua Ling, Yuan Jiang, Li-Juan Liu, Chen Liang, Li-Rong Dai

Abstract

This paper presents methods of making use of text supervision to improve the performance of sequence-to-sequence (seq2seq) voice conversion. Compared with conventional frame-to-frame voice conversion approaches, the seq2seq acoustic modeling method proposed in our previous work achieved higher naturalness and similarity. In this paper, we further improve its performance by utilizing the text transcriptions of parallel training data. First, a multi-task learning structure is designed which adds auxiliary classifiers to the middle layers of the seq2seq model and predicts linguistic labels as a secondary task. Second, a data-augmentation method is proposed which utilizes text alignment to produce extra parallel sequences for model training. Experiments are conducted to evaluate our proposed method with training sets at different sizes. Experimental results show that the multi-task learning with linguistic labels is effective at reducing the errors of seq2seq voice conversion. The data-augmentation method can further improve the performance of seq2seq voice conversion when only 50 or 100 training utterances are available.

Introduction

Sequence-to-sequence (seq2seq) models have revolutionized voice conversion (VC) by eliminating the need for strict frame-by-frame alignment between source and target speakers. However, these models often suffer from instability, leading to "stuttering," skipped syllables, or mispronunciations. These issues arise because the models learn purely from acoustic features, and the attention mechanism can easily lose track of the linguistic content. This paper addresses these challenges by re-introducing text transcriptions into the training process as a form of supervision to stabilize the conversion process, particularly in low-resource scenarios.

Proposed Methods

Multi-Task Learning (MTL) with Linguistic Labels

The core of the proposed improvement is a multi-task learning framework that forces the model to maintain a clear representation of the spoken content. As shown in the architecture below, auxiliary classifiers (AC) are integrated into the encoder's output and the decoder's input. During training, these classifiers are tasked with predicting linguistic labels, such as phonemes and tones, alongside the primary acoustic conversion task. This dual-objective approach acts as a regularizer, ensuring the hidden layers capture robust linguistic features that guide the attention mechanism.

Proposed seq2seq architecture with auxiliary classifiers for multi-task learning.

Data Augmentation (DA) by Text Alignment

To improve performance when training data is scarce, the authors propose a data augmentation method based on text-to-speech alignment. By identifying silence points between words, long sentences are sliced into multiple smaller parallel fragments. This process effectively multiplies the training samples and forces the model to learn granular relationships between source and target voices, rather than over-fitting to specific long-sentence contexts.

Experimental Results

Objective Evaluation

The proposed methods were evaluated using a Mandarin parallel corpus with training sets ranging from 50 to 1000 utterances. Objective metrics, including Mel-cepstrum distortion (MCD) and Fundamental Frequency Root Mean Square Error ($F_0$ RMSE), demonstrate significant improvements. As shown in the table below, the combination of MTL and DA (Seq2seq-MT-DA) consistently outperforms the baseline, especially in low-resource settings (e.g., 50 or 100 utterances).

Objective evaluation results (MCD and F0 RMSE) across different training data sizes.

Subjective Evaluation and Stability

Subjective tests confirmed that text supervision dramatically reduces pronunciation errors. In the Male-to-Female task using 1000 training utterances, the number of utterances containing skips or repeats dropped from 12 in the baseline to just 3 with the MTL approach. Preference tests also indicated that listeners favored the proposed method for its superior naturalness and speaker similarity, particularly when training data was limited.

Conclusion

By incorporating text supervision through multi-task learning and data augmentation, the stability and quality of seq2seq voice conversion are significantly enhanced. While data augmentation is most beneficial in low-resource scenarios, multi-task learning with linguistic labels provides a consistent performance boost across all dataset sizes by grounding acoustic modeling in linguistic reality.

Reference: Zhang, J. X., Ling, Z. H., Jiang, Y., Liu, L. J., Liang, C., & Dai, L. R. (2019). Improving Sequence-to-Sequence Voice Conversion by Adding Text-Supervision. arXiv:1811.08111.

Summary updated

Abstract

This paper presents methods of making using of text supervision to improve the performance of sequence-to-sequence (seq2seq) voice conversion. Compared with conventional frame-to-frame voice conversion approaches, the seq2seq acoustic modeling method proposed in our previous work achieved higher naturalness and similarity. In this paper, we further improve its performance by utilizing the text transcriptions of parallel training data. First, a multi-task learning structure is designed which adds auxiliary classifiers to the middle layers of the seq2seq model and predicts linguistic labels as a secondary task. Second, a data-augmentation method is proposed which utilizes text alignment to produce extra parallel sequences for model training. Experiments are conducted to evaluate our proposed method with training sets at different sizes. Experimental results show that the multi-task learning with linguistic labels is effective at reducing the errors of seq2seq voice conversion. The data-augmentation method can further improve the performance of seq2seq voice conversion when only 50 or 100 training utterances are available.
BibTeX
@inproceedings{zhang2019improving,
  author = {Jing-Xuan Zhang and Zhen-Hua Ling and Yuan Jiang and Li-Juan Liu and Chen Liang and Li-Rong Dai},
  title = {{Improving Sequence-to-Sequence Acoustic Modeling by Adding Text-Supervision}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2019},
  url = {https://openreview.net/forum?id=w4sHr23z7a},
}
Figure