This paper, presented at ICASSP 2019, addresses a common frustration in sequence-to-sequence (seq2seq) voice conversion: the tendency for models to "stutter," skip syllables, or mispronounce words despite sounding otherwise natural. By re-introducing text transcriptions into the training process, the authors significantly stabilize the conversion process, especially when training data is scarce.
TL;DR
The authors improve sequence-to-sequence voice conversion by using text transcriptions as a "guide" during training. By forcing the model to recognize linguistic labels (multi-task learning) and chopping long sentences into smaller fragments (data augmentation), they significantly reduce pronunciation errors and improve performance on small datasets.
What Problem Does This Paper Address?
Voice Conversion (VC) aims to change the speaker's identity while preserving the spoken content. Modern seq2seq models (like the authors' previous SCENT framework) are popular because they don't require frame-by-frame alignment between the source and target speakers. They use an "attention mechanism" to decide which part of the source audio corresponds to the target audio.
However, these models are notoriously unstable. Because they learn purely from acoustic features (audio-to-audio), the attention mechanism can "get lost," leading to:
- Dropped phonemes: Skipping a sound or word.
- Repeating phonemes: Getting stuck in a loop.
- Mispronunciations: Distorting the linguistic content.
The authors argue that since we usually have the text transcriptions for parallel training data anyway, we should use that text to "supervise" the model and keep it grounded in the actual words being spoken.
Methods
The authors propose two main enhancements to their baseline SCENT (Seq2seq ConvErsion NeTwork) architecture:
-
Multi-Task Learning (MTL) with Linguistic Labels: The authors add "auxiliary classifiers" to the middle of the network (specifically at the encoder's output and the decoder's input). During training, these classifiers are tasked with predicting the linguistic labels (phonemes and tones) of the current segment. This forces the hidden layers of the neural network to maintain a clear representation of what is being said, which "steers" the model away from acoustic hallucinations or skips.
-
Data Augmentation (DA) by Text Alignment: Seq2seq models usually require a lot of data to generalize well. To fix this, the authors use text-to-speech alignment to find "alignment points" (usually silences between words). They then slice the long sentences into many smaller parallel fragments. For a sentence with $N$ silence points, they can generate $C_N^2$ different fragments. This effectively multiplies the amount of training data and helps the model learn shorter, more robust mappings.
Datasets
- Mandarin Parallel Corpus: 1060 parallel utterances from one male and one female speaker.
- Setup: 1000 utterances for training, 30 for validation, and 30 for testing.
- Low-Resource Testing: To test the robustness of their data augmentation, they created smaller training subsets containing only 50, 100, 200, and 400 utterances.
Key Insights and Ideas
- Text as a Regularizer: The most important "aha" moment is that linguistic supervision acts as a powerful regularizer. By making the model "prove" it knows what phoneme it is processing, the attention mechanism becomes much more stable.
- Fragment-based Learning: Training on fragments rather than just whole sentences prevents the model from over-relying on the specific context of a long recording. It forces the model to learn the fundamental relationship between source and target voices at a more granular level.
- Solving the "Small Data" Problem: The data augmentation method is particularly effective when you have fewer than 100 recordings. This is a common real-world constraint where a target speaker might only provide a few minutes of audio.
Key Results
- Stability: In subjective tests, the number of "mistakes" (skips/repeats) dropped dramatically. For example, in the Male-to-Female task with 1000 training utterances, mistakes dropped from 12 to 3.
- Objective Metrics: Using the MTL+DA approach with only 50 training utterances achieved a Mel-cepstrum distortion (MCD) of 4.473 dB, significantly beating the baseline's 5.116 dB.
- Subjective Preference: In ABX preference tests (where listeners choose which audio sounds better/more like the target), the proposed method was preferred over the baseline by a wide margin, especially in the 50-utterance "low-resource" scenario.
Limitations and Future Work
- Diminishing Returns on Large Data: The authors noted that when they had the full 1000-utterance dataset, the Data Augmentation (DA) provided little to no benefit over the Multi-Task Learning (MTL) alone. This suggests DA is primarily a tool for low-resource scenarios.
- Parallel Data Requirement: The method still relies on parallel data (both speakers saying the same thing). Future work could explore applying these text-supervision techniques to non-parallel voice conversion.
- Vocoder Dependency: The quality is still somewhat tied to the WaveNet vocoder used for final synthesis, which is computationally expensive.
Reference: Zhang, J. X., Ling, Z. H., Jiang, Y., Liu, L. J., Liang, C., & Dai, L. R. (2019). Improving Sequence-to-Sequence Voice Conversion by Adding Text-Supervision. arXiv:1811.08111.
Summary updated