TL;DR
This paper introduces SCENT, a sequence-to-sequence neural network for voice conversion that eliminates the need for strict frame-by-frame alignment. By combining Mel-spectrograms with ASR bottleneck features and using an attention mechanism, the model jointly converts pitch, timbre, and—crucially—speech duration. Paired with a WaveNet vocoder, it significantly outperforms traditional frame-based baselines in both objective metrics and human evaluations.
What Problem Does This Paper Address?
Traditional voice conversion (VC) pipelines rely heavily on frame-by-frame alignment algorithms, like Dynamic Time Warping (DTW), to pair source and target speech during training. Because the acoustic models are built on this strict frame-by-frame basis, the converted speech is forced to have the exact same duration and timing as the source utterance.
However, human speech is highly dynamic; different speakers have different speaking rates, pauses, and rhythms. Forcing a target speaker's voice into the source speaker's exact timing constraints limits the naturalness and similarity of the generated audio. Additionally, older methods often model spectral (vocal tract) features and fundamental frequency ($F_0$, or pitch) separately, which can lead to disjointed and robotic-sounding results.
Methods
The authors propose the Sequence-to-sequence ConvErsion NeTwork (SCENT), which adapts the encoder-decoder architecture (popularized in machine translation and Text-to-Speech) for Voice Conversion.
- Inputs: Instead of traditional vocoder features, the model uses Mel-spectrograms. Crucially, these are concatenated with bottleneck features extracted from a pre-trained Automatic Speech Recognition (ASR) model. This injects explicit linguistic context into the acoustic data.
- Encoder: Uses a pyramid Bidirectional LSTM (pBLSTM) to downsample the input sequence. This shorter sequence makes it much easier for the attention mechanism to converge. They also inject a sinusoidal "location code" to help the model track its position in the sequence.
- Attention: A hybrid, location-aware attention mechanism is used. It is modified with a "forward attention" constraint to enforce the monotonic, left-to-right nature of human speech.
- Decoder & Vocoder: An auto-regressive LSTM predicts the target Mel-spectrograms and a "stop token" (completion probability) step-by-step. A PostNet (CNN) refines the output, and finally, a speaker-dependent WaveNet vocoder synthesizes the raw, high-fidelity audio waveform from the predicted spectrograms.
Datasets
- Mandarin Dataset (Internal): 1,060 parallel utterances from one male and one female speaker.
- CMU ARCTIC (English): 1,132 parallel utterances from one male (rms) and one female (slt) speaker.
- ASR Training Data (Internal): ~10,000 hours of Mandarin and ~3,000 hours of English speech. This massive dataset was used only to pre-train the ASR model that extracts the linguistic bottleneck features.
Key Insights and Ideas
- Duration Conversion is a Game Changer: By ditching DTW and using an attention mechanism, the model naturally learns to adjust the speaking rate. A fast-speaking male source can be converted into a slower-speaking female target with natural pacing, rather than sounding artificially rushed.
- ASR Features are Non-Negotiable for Seq2Seq VC: Ablation studies showed that without the ASR bottleneck features, the model suffered from severe mispronunciations. Acoustic Mel-spectrograms alone aren't enough for the seq2seq model to maintain linguistic stability; it needs the ASR model to "tell" it what phonemes are being spoken.
- Joint Modeling Beats Separate Modeling: Predicting Mel-spectrograms directly allows the network to handle pitch ($F_0$) and vocal tract (spectral) features simultaneously, avoiding the artifacts introduced by traditional, separate pitch-shifting algorithms.
Key Results
- Objective Metrics: SCENT achieved lower Mel-cepstral distortion (MCD) and $F_0$ Root Mean Square Error (RMSE) than baselines (GMM, DNN, and the authors' own top-tier VCC2018 model). For English Female-to-Male conversion, it hit an $F_0$ RMSE of 9.899 Hz (vs. 11.116 Hz for the VCC2018 baseline).
- Duration Accuracy: The average absolute difference in duration (DDUR) between the converted speech and the ground-truth target speech dropped massively. For Mandarin Female-to-Male, DDUR was just 0.194 seconds, compared to 1.147 seconds for baselines lacking duration modification.
- Subjective Quality (MOS): In human evaluations, SCENT scored between 3.66 and 4.10 out of 5 for both naturalness and similarity across both languages, consistently outperforming all baseline methods.
Limitations and Future Work
- Attention Instability: Like many seq2seq models, SCENT occasionally suffers from alignment errors. The authors noted instances of skipped phonemes, repeated words, or premature stopping during inference.
- Data Hungry: The model has ~7.5 million parameters and requires a substantial amount of parallel training data. It can struggle to generalize to completely unseen utterances.
- ASR Dependency: The system's pronunciation accuracy is entirely bottlenecked by the quality of the external ASR model used for feature extraction.
- Future Work: The authors plan to investigate ways to improve the stability of the attention calculation to reduce conversion errors, and to explore how varying the training set size impacts overall performance.
Summary updated