Abstract
This paper presents a novel neural network architecture, the Sequence-to-sequence ConvErsion NeTwork (SCENT), for acoustic modeling in voice conversion. Unlike traditional methods that rely on strict frame-by-frame alignment, SCENT utilizes an attention mechanism to implicitly align feature sequences between source and target speakers. By adopting Mel-scale spectrograms as acoustic features and incorporating bottleneck features from a pre-trained automatic speech recognition (ASR) model, the system jointly converts pitch, timbre, and speech duration. A WaveNet vocoder is then used to reconstruct high-fidelity waveforms. Experimental results demonstrate that SCENT achieves superior objective and subjective performance compared to baseline GMM and DNN models, particularly in its ability to perform natural duration conversion.
Introduction
Traditional voice conversion (VC) pipelines are often limited by their reliance on frame-by-frame alignment algorithms, such as Dynamic Time Warping (DTW). These methods force the converted speech to strictly match the duration and timing of the source utterance, which fails to capture the natural rhythmic variations between different speakers. Furthermore, modeling spectral features and fundamental frequency ($F_0$) separately can lead to robotic and disjointed audio quality.
To address these limitations, we propose SCENT, a sequence-to-sequence framework that eliminates the need for explicit frame alignment and enables the unified conversion of acoustic features and durations.
The SCENT Framework
The SCENT system architecture is designed to handle the complex mapping between source and target speech by leveraging linguistic context and flexible temporal modeling.
Input Features and Encoder
The model takes Mel-spectrograms as primary acoustic inputs, which are concatenated with bottleneck features extracted from a pre-trained ASR model. This combination ensures that the network has access to both fine-grained acoustic data and high-level linguistic context. The encoder utilizes a pyramid Bidirectional LSTM (pBLSTM) to downsample the input sequence, facilitating more efficient attention convergence. A sinusoidal "location code" is also injected to provide the model with positional awareness.
Attention and Decoder
SCENT employs a hybrid, location-aware attention mechanism modified with a "forward attention" constraint to enforce the monotonic nature of speech. The decoder is an auto-regressive LSTM that predicts target Mel-spectrograms and a "stop token" step-by-step. A convolutional PostNet further refines these outputs before they are passed to a speaker-dependent WaveNet vocoder for final waveform synthesis.
Experimental Evaluation
Datasets
The model was evaluated on two primary datasets:
- Mandarin Dataset: 1,060 parallel utterances from one male and one female speaker.
- CMU ARCTIC (English): 1,132 parallel utterances from one male (rms) and one female (slt) speaker.
- ASR Pre-training: Approximately 13,000 hours of combined Mandarin and English speech were used to train the auxiliary ASR model for bottleneck feature extraction.
Key Results
- Duration Accuracy: SCENT significantly improved duration conversion. For Mandarin Female-to-Male conversion, the average absolute difference in duration (DDUR) was reduced to 0.194 seconds, compared to over 1.1 seconds for traditional baselines.
- Objective Metrics: The proposed method achieved lower Mel-cepstral distortion (MCD) and $F_0$ Root Mean Square Error (RMSE) than GMM, DNN, and previous state-of-the-art models.
- Subjective Quality: In Mean Opinion Score (MOS) tests, SCENT consistently outperformed baselines in both naturalness and speaker similarity, scoring between 3.66 and 4.10 across various tasks.
Conclusion
SCENT represents a significant step forward in voice conversion by moving beyond frame-based constraints. The integration of ASR bottleneck features proved essential for linguistic stability, while the sequence-to-sequence architecture allowed for natural pacing and joint modeling of pitch and timbre. Future work will focus on improving the stability of the attention mechanism to further reduce occasional alignment errors such as skipped or repeated phonemes.
Summary updated