Sequence-to-Sequence Acoustic Modeling for Voice Conversion

Jing-Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, Li-Rong Dai

TL;DR

This paper introduces SCENT, a sequence-to-sequence neural network for voice conversion that eliminates the need for strict frame-by-frame alignment. By combining Mel-spectrograms with ASR bottleneck features and using an attention mechanism, the model jointly converts pitch, timbre, and—crucially—speech duration. Paired with a WaveNet vocoder, it significantly outperforms traditional frame-based baselines in both objective metrics and human evaluations.

What Problem Does This Paper Address?

Traditional voice conversion (VC) pipelines rely heavily on frame-by-frame alignment algorithms, like Dynamic Time Warping (DTW), to pair source and target speech during training. Because the acoustic models are built on this strict frame-by-frame basis, the converted speech is forced to have the exact same duration and timing as the source utterance.

However, human speech is highly dynamic; different speakers have different speaking rates, pauses, and rhythms. Forcing a target speaker's voice into the source speaker's exact timing constraints limits the naturalness and similarity of the generated audio. Additionally, older methods often model spectral (vocal tract) features and fundamental frequency ($F_0$, or pitch) separately, which can lead to disjointed and robotic-sounding results.

Methods

The authors propose the Sequence-to-sequence ConvErsion NeTwork (SCENT), which adapts the encoder-decoder architecture (popularized in machine translation and Text-to-Speech) for Voice Conversion.

  • Inputs: Instead of traditional vocoder features, the model uses Mel-spectrograms. Crucially, these are concatenated with bottleneck features extracted from a pre-trained Automatic Speech Recognition (ASR) model. This injects explicit linguistic context into the acoustic data.
  • Encoder: Uses a pyramid Bidirectional LSTM (pBLSTM) to downsample the input sequence. This shorter sequence makes it much easier for the attention mechanism to converge. They also inject a sinusoidal "location code" to help the model track its position in the sequence.
  • Attention: A hybrid, location-aware attention mechanism is used. It is modified with a "forward attention" constraint to enforce the monotonic, left-to-right nature of human speech.
  • Decoder & Vocoder: An auto-regressive LSTM predicts the target Mel-spectrograms and a "stop token" (completion probability) step-by-step. A PostNet (CNN) refines the output, and finally, a speaker-dependent WaveNet vocoder synthesizes the raw, high-fidelity audio waveform from the predicted spectrograms.

Datasets

  • Mandarin Dataset (Internal): 1,060 parallel utterances from one male and one female speaker.
  • CMU ARCTIC (English): 1,132 parallel utterances from one male (rms) and one female (slt) speaker.
  • ASR Training Data (Internal): ~10,000 hours of Mandarin and ~3,000 hours of English speech. This massive dataset was used only to pre-train the ASR model that extracts the linguistic bottleneck features.

Key Insights and Ideas

  • Duration Conversion is a Game Changer: By ditching DTW and using an attention mechanism, the model naturally learns to adjust the speaking rate. A fast-speaking male source can be converted into a slower-speaking female target with natural pacing, rather than sounding artificially rushed.
  • ASR Features are Non-Negotiable for Seq2Seq VC: Ablation studies showed that without the ASR bottleneck features, the model suffered from severe mispronunciations. Acoustic Mel-spectrograms alone aren't enough for the seq2seq model to maintain linguistic stability; it needs the ASR model to "tell" it what phonemes are being spoken.
  • Joint Modeling Beats Separate Modeling: Predicting Mel-spectrograms directly allows the network to handle pitch ($F_0$) and vocal tract (spectral) features simultaneously, avoiding the artifacts introduced by traditional, separate pitch-shifting algorithms.

Key Results

  • Objective Metrics: SCENT achieved lower Mel-cepstral distortion (MCD) and $F_0$ Root Mean Square Error (RMSE) than baselines (GMM, DNN, and the authors' own top-tier VCC2018 model). For English Female-to-Male conversion, it hit an $F_0$ RMSE of 9.899 Hz (vs. 11.116 Hz for the VCC2018 baseline).
  • Duration Accuracy: The average absolute difference in duration (DDUR) between the converted speech and the ground-truth target speech dropped massively. For Mandarin Female-to-Male, DDUR was just 0.194 seconds, compared to 1.147 seconds for baselines lacking duration modification.
  • Subjective Quality (MOS): In human evaluations, SCENT scored between 3.66 and 4.10 out of 5 for both naturalness and similarity across both languages, consistently outperforming all baseline methods.

Limitations and Future Work

  • Attention Instability: Like many seq2seq models, SCENT occasionally suffers from alignment errors. The authors noted instances of skipped phonemes, repeated words, or premature stopping during inference.
  • Data Hungry: The model has ~7.5 million parameters and requires a substantial amount of parallel training data. It can struggle to generalize to completely unseen utterances.
  • ASR Dependency: The system's pronunciation accuracy is entirely bottlenecked by the quality of the external ASR model used for feature extraction.
  • Future Work: The authors plan to investigate ways to improve the stability of the attention calculation to reduce conversion errors, and to explore how varying the training set size impacts overall performance.

Summary updated

Abstract

In this paper, a neural network named sequence-to-sequence ConvErsion NeTwork (SCENT) is presented for acoustic modeling in voice conversion. At training stage, a SCENT model is estimated by aligning the feature sequences of source and target speakers implicitly using attention mechanism. At the conversion stage, acoustic features and durations of source utterances are converted simultaneously using the unified acoustic model. Mel-scale spectrograms are adopted as acoustic features, which contain both excitation and vocal tract descriptions of speech signals. The bottleneck features extracted from source speech using an automatic speech recognition model are appended as an auxiliary input. A WaveNet vocoder conditioned on Mel-spectrograms is built to reconstruct waveforms from the outputs of the SCENT model. It is worth noting that our proposed method can achieve appropriate duration conversion, which is difficult in conventional methods. Experimental results show that our proposed method obtained better objective and subjective performance than the baseline methods using Gaussian mixture models and deep neural networks as acoustic models. This proposed method also outperformed our previous work, which achieved the top rank in Voice Conversion Challenge 2018. Ablation tests further confirmed the effectiveness of several components in our proposed method.
BibTeX
@article{zhang2019sequence,
  author = {Jing-Xuan Zhang and Zhen-Hua Ling and Li-Juan Liu and Yuan Jiang and Li-Rong Dai},
  title = {{Sequence-to-Sequence Acoustic Modeling for Voice Conversion}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=Ic7Bml1rkA},
}

Loading related content...

Figure