Abstract
This paper presents methods for leveraging text supervision to enhance the performance of sequence-to-sequence (seq2seq) voice conversion (VC). While seq2seq acoustic modeling offers superior naturalness and similarity compared to traditional frame-to-frame approaches, it often suffers from stability issues such as mispronunciations and phoneme skipping. We propose two primary enhancements: a multi-task learning (MTL) framework that predicts linguistic labels via auxiliary classifiers, and a data-augmentation method utilizing text alignment to generate additional parallel training fragments. Experimental results demonstrate that these methods significantly reduce pronunciation errors and improve objective speech metrics, particularly in low-resource scenarios with as few as 50 training utterances.
Introduction
Voice Conversion (VC) technology aims to transform the speaker identity of a source utterance while preserving its linguistic content. Traditional frame-to-frame mapping often results in robotic speech due to its inability to model prosodic variations effectively. Seq2seq models address this by learning to adjust duration and rhythm. However, the reliance on attention mechanisms can lead to "stability" problems, where the model loses its place in the sequence, causing stutters, skips, or garbled words. This research explores how text transcriptions, which are typically available for parallel training data, can be used as a supervisory signal to regularize the model and ensure linguistic consistency.
Proposed Methodology
Multi-task Learning with Linguistic Labels
To stabilize the attention mechanism, we introduce auxiliary classifiers (AC) into the seq2seq architecture. These classifiers are attached to the encoder's output and the decoder's input, tasked with predicting phonemes and tones (linguistic labels) as a secondary objective. This forces the network's hidden layers to maintain a robust linguistic representation, acting as a constraint that prevents the model from deviating from the intended text during conversion.
Data Augmentation via Text Alignment
Seq2seq models are notoriously data-hungry. To improve performance in low-resource settings, we propose a data-augmentation technique based on silence-based slicing. By using text-to-speech alignment to identify pauses, we can segment long parallel utterances into multiple shorter, high-quality fragments. This process effectively multiplies the training samples without introducing the co-articulation artifacts that would occur if slicing were performed mid-word.
Experimental Evaluation
Setup
The system was evaluated using a Mandarin Chinese dataset consisting of 1,060 parallel utterances from one male and one female speaker. We tested the model across various training set sizes (50, 100, 200, 400, and 1,000 utterances) to assess its robustness in low-resource conditions. Features included 80-dimensional mel-spectrograms and 512-dimensional bottleneck features from a pre-trained speech recognition model.
Results and Discussion
Objective evaluations using Mel-cepstrum distortion (MCD) and F0 Root Mean Square Error (RMSE) show that the proposed text-supervised models consistently outperform the baseline seq2seq system. The multi-task learning approach (Seq2seq-MT) provides a baseline improvement in stability, while the data-augmentation (Seq2seq-MT-DA) offers substantial gains when training data is extremely limited (50–100 utterances).
Subjective listening tests further confirmed these findings. Pronunciation errors dropped significantly; for instance, in the male-to-female task with 1,000 utterances, the number of mistakes fell from 12 in the baseline to just 3 in the proposed model. ABX preference tests indicated that listeners overwhelmingly preferred the naturalness and speaker similarity of the text-supervised models, especially in the 50-utterance scenario.
Conclusion
By integrating text supervision through multi-task learning and data augmentation, we have significantly improved the stability and quality of seq2seq voice conversion. These techniques are particularly effective for low-resource speaker adaptation. Future work will focus on applying these methods to even smaller datasets and exploring cross-lingual voice conversion applications.
Summary updated