TL;DR
This paper introduces a novel "Dynamic Model Selection" (DMS) approach for spectral voice conversion. Instead of using a single, fixed model, DMS dynamically selects the best model from a pre-defined set at each frame of the source speech, adapting to the specific acoustic features. This leads to improved conversion quality and a closer match to the target speaker's voice.
What Problem Does This Paper Address?
Voice conversion (VC) aims to modify a source speaker's voice to sound like a target speaker. A common challenge in VC is achieving high-quality converted speech that is both perceptually close to the target and natural-sounding. Traditional statistical VC methods often rely on a single, static model trained to represent an average mapping between source and target features. However, this single model might not be optimal for all acoustic conditions within a speech utterance. For instance, stable vowel sounds might benefit from a complex, precise model, while rapid transitions or less common sounds might require a simpler, more general model to avoid overfitting and artifacts. This paper addresses the limitation of static models by proposing a dynamic approach that adapts the model choice on-the-fly.
Methods
The core innovation is the Dynamic Model Selection (DMS). The authors first build a set of potential "best" models during a training phase. This set includes Gaussian Mixture Models (GMMs) and, importantly, Mixtures of Probabilistic Principal Component Analyzers (PPCAs). PPCAs are highlighted as a way to model covariance structures more flexibly than standard GMMs, allowing for a range of models with increasing complexity.
During the conversion phase, for each incoming frame of source speech, the DMS method calculates the likelihood of that frame's acoustic features under each model in the pre-selected set. The model that best explains the current source frame's features is then chosen for the conversion of that specific frame. This allows the system to use simpler, more general models for acoustically "difficult" or transitional frames and more complex, precise models for stable frames, effectively balancing model complexity with acoustic context.
The paper also details the use of Line Spectral Frequencies (LSFs) as spectral features and mentions the Mel-frequency based True Envelope (MTE) method for spectral envelope estimation.
Datasets
The study uses three corpora of French speech recorded by three speakers with different accents:
- Speaker S: Standard French accent (used as the source voice).
- Speaker A: Hispanic accent (used as a target voice).
- Speaker B: French Canadian accent (used as a target voice).
Each corpus consists of approximately 9-10 minutes of phonetically balanced utterances. The corpora were split into training (190 utterances) and testing (10 utterances) sets. A pre-rejecting method was applied to remove mismatched source-target frames from the training data, which improved conversion quality.
Key Insights and Ideas
- Dynamic Adaptation is Key: The central idea is that a single, static model is insufficient for optimal voice conversion across an entire utterance. Adapting the model choice dynamically to the source speech's acoustic characteristics is crucial for improving conversion quality.
- PPCAs for Flexible Modeling: Mixtures of PPCAs offer a more flexible way to model covariance structures compared to standard GMMs, enabling the creation of a diverse set of models with varying complexity.
- Model Selection Based on Likelihood: Using the likelihood of the source frame under each candidate model as the selection criterion is an intuitive and effective way to dynamically choose the most appropriate model.
- Trade-off Between Complexity and Accuracy: The DMS approach explicitly manages the trade-off between model complexity and accuracy. Simpler models are used for less predictable frames, while more complex models are employed for more stable, predictable frames.
Key Results
Subjective evaluations were conducted with 27 listeners.
- Conversion Effect: For both target speakers (A and B), the DMS method showed a slight improvement over the baseline GMM16 method in terms of perceived conversion effect (i.e., how close the converted voice sounded to the target). Mean vote values were 2.56 for DMS vs. 2.32 for GMM16 for speaker A, and 2.22 for DMS vs. 2.17 for GMM16 for speaker B.
- Converted Speech Quality: A comparison category rating test revealed that the DMS method provided significantly better speech quality than the GMM16 baseline. The preference mean score for DMS was 0.31 for speaker A and 0.36 for speaker B, with statistical analysis confirming the significance of these differences.
Limitations and Future Work
- Limitations: The paper notes that a "third-speaker effect" (where listeners couldn't clearly identify either the source or target speaker) was observed, which is a known issue in voice conversion, especially for intra-gender tasks. The presence of reverberation in the target speaker B's recordings might have influenced the results for that speaker.
- Future Work: The authors plan to investigate more appropriate criteria for selecting the best model during conversion. They also intend to apply the DMS framework to other advanced conversion methods, such as the one described in [8] which incorporates dynamic features and global variance, to further improve synthesis quality.
Summary updated