Using a Manifold Vocoder for Spectral Voice and Style Conversion

Tuan Dinh, Alexander Kain, Kris TjadenInterspeech 2019 · DOI

Abstract

This paper introduces a manifold vocoder based on a Variational Autoencoder (VAE) designed for spectral voice and style conversion. By representing speech in a low-dimensional, continuous latent space (VAE-12), the system overcomes the interpolation artifacts common in traditional features like Mel-Cepstral Coefficients (MCEPs). Experimental results demonstrate that VAE-12 provides superior vocoding quality and voice conversion performance compared to standard baselines. Furthermore, the approach significantly improves speech intelligibility for individuals with Parkinson’s disease, nearly doubling keyword recall in specific cases.

Introduction

In speech processing, the choice of acoustic features is critical for the quality of synthesized or converted speech. Traditional features, such as Mel-Cepstral Coefficients (MCEPs), are widely used but lack a natural manifold structure. When machine learning models perform regression or averaging in these feature spaces, the resulting "in-between" points often correspond to physically impossible or muffled sounds. This paper proposes a "Manifold Vocoder" that utilizes a VAE to learn a smooth, continuous representation of speech spectra, ensuring that interpolated points remain on the manifold of valid human speech.

Methodology

The core of the proposed system is a speaker-independent VAE trained on the TIMIT dataset. The encoder compresses a 39-dimensional spectral vector into a 12-dimensional latent space ($z$). This VAE-12 representation is then used for two primary tasks:

  1. Voice Conversion (VC): A Deep Neural Network (DNN) maps VAE features from a source speaker to a target speaker.
  2. Style Conversion: To transform "habitual" speech into a "clear" style, the authors employ a custom skip-connection DNN architecture. This design allows the model to focus on learning the spectral differences between styles rather than reconstructing the entire signal, facilitating more effective style adaptation.
Architecture of the skip-connection DNN used for style conversion.

Experimental Evaluation

The system was evaluated using several datasets, including TIMIT for initial training, the Voice Conversion Challenge (VCC) 2016 for benchmarking, and the Harvard Sentences Database for clinical applications involving speakers with Parkinson’s Disease (PD) and Multiple Sclerosis (MS).

Vocoding and Voice Conversion Quality

Perceptual listening tests were conducted to compare VAE-12 against traditional features, including MCEP-12, MCEP-40, and 20th-order Line Spectral Frequencies (LSF-20).

Perceptual evaluation results for (a) vocoding quality and (b) voice conversion performance across different feature representations.

As shown in Figure 5, VAE-12 consistently outperformed MCEP-12 in naturalness. In voice conversion tasks (Figure 5b), VAE-12 was found to be the closest to natural speech among all tested features, highlighting the benefits of the smooth latent space for mapping tasks. The compactness of the 12-dimensional VAE feature proved more efficient at capturing the essence of speech than higher-dimensional traditional features.

Intelligibility Improvement

The most significant finding was the system's ability to enhance the speech of a speaker with Parkinson’s Disease (PDM6). By converting habitual speech to a clearer style using the manifold vocoder, keyword recall accuracy for listeners improved from 24% to 46%. This nearly twofold increase demonstrates the potential of manifold-based representations for assistive speech technologies.

Conclusion

The manifold vocoder provides a robust framework for high-quality speech conversion by ensuring that acoustic features remain within a valid speech manifold. While the system showed impressive gains in intelligibility for some speakers, performance varied across individuals, likely due to the diverse strategies speakers use to achieve "clear" speech. Future research will focus on many-to-many voice conversion and developing more generalized style mappings that can improve intelligibility without requiring extensive speaker-specific data.

Summary updated

BibTeX
@inproceedings{dinh2019using,
  author = {Tuan Dinh and Alexander Kain and Kris Tjaden},
  title = {{Using a Manifold Vocoder for Spectral Voice and Style Conversion}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1176},
}
Figure