Non-parallel training for voice conversion by maximum likelihood constrained adaptation

Athanasios Mouchtaris, Jan Van der Spiegel, Paul Mueller

TL;DR

This paper introduces a novel voice conversion method that bypasses the need for parallel speech corpora (where the same sentences are spoken by both source and target speakers). By adapting pre-trained conversion parameters from a parallel corpus to a non-parallel one, they achieve significant error reduction, making voice conversion more practical when parallel data is unavailable.

What Problem Does This Paper Address?

Voice conversion aims to make one speaker's voice sound like another's. Traditional methods rely on "parallel corpora" – recordings of the exact same sentences spoken by both the source and target speakers. This is crucial for aligning speech segments and estimating conversion parameters. However, collecting such parallel data is often difficult or impossible, especially for rare voices or when direct access to speakers is limited. This paper tackles the challenge of performing voice conversion when only non-parallel corpora are available for the desired speaker pair.

Methods

The core of the proposed method is Maximum Likelihood Constrained Adaptation. The authors leverage existing voice conversion techniques that work with parallel corpora to derive an initial conversion function. This function is based on modeling the joint probability distribution of spectral features from a source and target speaker using Gaussian Mixture Models (GMMs).

The innovation lies in adapting these pre-trained GMM parameters to a new, non-parallel corpus. This adaptation is achieved through a probabilistic linear transformation. Essentially, they model how the spectral features of the source and target speakers in the non-parallel corpus relate to the features from the parallel corpus using linear transformations. This allows them to "tune" the conversion function to the specific characteristics of the new speaker pair without requiring direct utterance alignment. The adaptation process itself is framed as a constrained maximum-likelihood estimation problem, solvable using the Expectation-Maximization (EM) algorithm.

Datasets

The primary dataset used for both training and evaluation is the VOICES corpus. This is a parallel corpus. The authors use portions of this corpus for both the "ideal" parallel training scenario and as the source for deriving initial conversion parameters that are then adapted to non-parallel scenarios. They also evaluate performance on different subsets of speakers from this corpus, referred to as "Test 1" and "Test 2".

Key Insights and Ideas

  • Breaking the Parallel Corpus Bottleneck: The most significant insight is demonstrating that high-quality voice conversion is achievable without the stringent requirement of parallel speech data. This dramatically expands the applicability of voice conversion technology.
  • Adaptation as a Bridge: The paper shows that adapting parameters derived from a parallel corpus to a non-parallel one is a viable and effective strategy. This "transfer learning" approach for voice conversion is a key contribution.
  • Linear Transformation for Adaptation: Using probabilistic linear transformations to bridge the gap between parallel and non-parallel data provides a mathematically tractable and effective solution.
  • Performance Trade-offs: While parallel training generally yields the best results, the adapted non-parallel method comes remarkably close, especially when sufficient adaptation data is available.

Key Results

The paper demonstrates that their adaptation method significantly reduces the conversion error compared to simply applying parameters trained on one speaker pair to a different, non-parallel pair.

  • Error Reduction: The adaptation technique can reduce the normalized mean-squared error by up to 30% in many cases, particularly when the initial spectral distance between speakers is large.
  • Comparable Performance: The performance of the adapted non-parallel method is shown to be comparable to the ideal case of training with a parallel corpus.
  • Adaptation Parameter Sensitivity: Figure 2(a) indicates that the error decreases significantly with an increasing number of adaptation parameters, but plateaus after around 4 parameters, suggesting an optimal range for modeling speaker statistics.
  • Dataset Size Impact: Figure 2(b) shows that increasing the size of the non-parallel corpus also leads to error reduction, with diminishing returns beyond 5,000-10,000 vectors.

Limitations and Future Work

  • Parallel Corpus Dependency: The method still relies on having some parallel corpus available to derive the initial conversion parameters. The ultimate goal of zero parallel data is not yet achieved.
  • Condition Mismatch: The authors acknowledge that if the parallel and non-parallel corpora are recorded under significantly different conditions (e.g., microphone quality, reverberation), the adaptation might not yield significant improvements. This is an area they intend to explore further.
  • Subjective Evaluation: While quantitative results are presented, the paper mentions that subjective listening tests were conducted and indicated satisfying performance, but these are not detailed in the provided text. Future work could involve more extensive subjective evaluations.
  • Further Analysis: The authors plan to further analyze the observed phenomenon where the initial distance is decreased (error less than one) even without adaptation, using a larger dataset.

Summary updated

Abstract

The objective of voice conversion methods is to modify the speech characteristics of a particular speaker in such manner, as to sound like speech by a different target speaker. Current voice conversion algorithms are based on deriving a conversion function by estimating its parameters through a corpus that contains the same utterances spoken by both speakers. Such a corpus, usually referred to as a parallel corpus, has the disadvantage that many times it is difficult or even impossible to collect. Here, we propose a voice conversion method that does not require a parallel corpus for training, i.e. the spoken utterances by the two speakers need not be the same, by employing speaker adaptation techniques to adapt to a particular pair of source and target speakers, the derived conversion parameters from a different pair of speakers. We show that adaptation reduces the error obtained when simply applying the conversion parameters of one pair of speakers to another by a factor that can reach 30% in many cases, and with performance comparable with the ideal case when a parallel corpus is available.
BibTeX
@inproceedings{mouchtaris2004non,
  author = {Athanasios Mouchtaris and Jan Van der Spiegel and Paul Mueller},
  title = {{Non-parallel training for voice conversion by maximum likelihood constrained adaptation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2004},
  doi = {10.1109/icassp.2004.1325907},
}
Figure