Sequence-to-Sequence Acoustic Modeling for Voice Conversion

Jing-Xuan Zhang, Zhen-Hua Ling, Li-Juan Liu, Yuan Jiang, Li-Rong Dai

Abstract

This paper presents a novel neural network architecture, the Sequence-to-sequence ConvErsion NeTwork (SCENT), for acoustic modeling in voice conversion. Unlike traditional methods that rely on strict frame-by-frame alignment, SCENT utilizes an attention mechanism to implicitly align feature sequences between source and target speakers. By adopting Mel-scale spectrograms as acoustic features and incorporating bottleneck features from a pre-trained automatic speech recognition (ASR) model, the system jointly converts pitch, timbre, and speech duration. A WaveNet vocoder is then used to reconstruct high-fidelity waveforms. Experimental results demonstrate that SCENT achieves superior objective and subjective performance compared to baseline GMM and DNN models, particularly in its ability to perform natural duration conversion.

Introduction

Traditional voice conversion (VC) pipelines are often limited by their reliance on frame-by-frame alignment algorithms, such as Dynamic Time Warping (DTW). These methods force the converted speech to strictly match the duration and timing of the source utterance, which fails to capture the natural rhythmic variations between different speakers. Furthermore, modeling spectral features and fundamental frequency ($F_0$) separately can lead to robotic and disjointed audio quality.

To address these limitations, we propose SCENT, a sequence-to-sequence framework that eliminates the need for explicit frame alignment and enables the unified conversion of acoustic features and durations.

The SCENT Framework

The SCENT system architecture is designed to handle the complex mapping between source and target speech by leveraging linguistic context and flexible temporal modeling.

Overall pipeline of the SCENT voice conversion system, showing the integration of ASR bottleneck features and the WaveNet vocoder.

Input Features and Encoder

The model takes Mel-spectrograms as primary acoustic inputs, which are concatenated with bottleneck features extracted from a pre-trained ASR model. This combination ensures that the network has access to both fine-grained acoustic data and high-level linguistic context. The encoder utilizes a pyramid Bidirectional LSTM (pBLSTM) to downsample the input sequence, facilitating more efficient attention convergence. A sinusoidal "location code" is also injected to provide the model with positional awareness.

Attention and Decoder

SCENT employs a hybrid, location-aware attention mechanism modified with a "forward attention" constraint to enforce the monotonic nature of speech. The decoder is an auto-regressive LSTM that predicts target Mel-spectrograms and a "stop token" step-by-step. A convolutional PostNet further refines these outputs before they are passed to a speaker-dependent WaveNet vocoder for final waveform synthesis.

Detailed architecture of the SCENT encoder and decoder, highlighting the attention mechanism and the use of location codes.

Experimental Evaluation

Datasets

The model was evaluated on two primary datasets:

  • Mandarin Dataset: 1,060 parallel utterances from one male and one female speaker.
  • CMU ARCTIC (English): 1,132 parallel utterances from one male (rms) and one female (slt) speaker.
  • ASR Pre-training: Approximately 13,000 hours of combined Mandarin and English speech were used to train the auxiliary ASR model for bottleneck feature extraction.

Key Results

  • Duration Accuracy: SCENT significantly improved duration conversion. For Mandarin Female-to-Male conversion, the average absolute difference in duration (DDUR) was reduced to 0.194 seconds, compared to over 1.1 seconds for traditional baselines.
  • Objective Metrics: The proposed method achieved lower Mel-cepstral distortion (MCD) and $F_0$ Root Mean Square Error (RMSE) than GMM, DNN, and previous state-of-the-art models.
  • Subjective Quality: In Mean Opinion Score (MOS) tests, SCENT consistently outperformed baselines in both naturalness and speaker similarity, scoring between 3.66 and 4.10 across various tasks.

Conclusion

SCENT represents a significant step forward in voice conversion by moving beyond frame-based constraints. The integration of ASR bottleneck features proved essential for linguistic stability, while the sequence-to-sequence architecture allowed for natural pacing and joint modeling of pitch and timbre. Future work will focus on improving the stability of the attention mechanism to further reduce occasional alignment errors such as skipped or repeated phonemes.

Summary updated

Abstract

In this paper, a neural network named Sequence-to-sequence ConvErsion NeTwork (SCENT) is presented for acoustic modeling in voice conversion. At training stage, a SCENT model is estimated by aligning the feature sequences of source and target speakers implicitly using attention mechanism. At conversion stage, acoustic features and durations of source utterances are converted simultaneously using the unified acoustic model. Mel-scale spectrograms are adopted as acoustic features which contain both excitation and vocal tract descriptions of speech signals. The bottleneck features extracted from source speech using an automatic speech recognition (ASR) model are appended as auxiliary input. A WaveNet vocoder conditioned on Mel-spectrograms is built to reconstruct waveforms from the outputs of the SCENT model. It is worth noting that our proposed method can achieve appropriate duration conversion which is difficult in conventional methods. Experimental results show that our proposed method obtained better objective and subjective performance than the baseline methods using Gaussian mixture models (GMM) and deep neural networks (DNN) as acoustic models. This proposed method also outperformed our previous work which achieved the top rank in Voice Conversion Challenge 2018. Ablation tests further confirmed the effectiveness of several components in our proposed method.
BibTeX
@article{zhang2019sequence,
  author = {Jing-Xuan Zhang and Zhen-Hua Ling and Li-Juan Liu and Yuan Jiang and Li-Rong Dai},
  title = {{Sequence-to-Sequence Acoustic Modeling for Voice Conversion}},
  journal = {IEEE ACM Trans. Audio Speech Lang. Process.},
  year = {2019},
  url = {https://openreview.net/forum?id=Ic7Bml1rkA},
}
Figure