Alleviating the one-to-many mapping problem in voice conversion with context-dependent modeling

Elizabeth Godoy, Olivier Rosec, Thierry ChonavelInterspeech 2009

TL;DR

This paper tackles the "one-to-many" mapping problem in voice conversion (VC), where a single source speech segment can correspond to multiple target speech variations, leading to degraded conversion quality. The authors propose a novel context-dependent modeling approach using phonetic information within Gaussian Mixture Models (GMMs) to significantly improve spectral conversion accuracy.

What Problem Does This Paper Address?

Voice Conversion (VC) aims to make one person's speech sound like another's. A common technique involves modeling the acoustic features of speech, like spectral envelopes. A significant challenge in this field is the "one-to-many" mapping problem. This arises because even when speech segments are aligned in time (e.g., the same phoneme), the acoustic realization can differ significantly between speakers. A GMM-based VC system, trained on acoustically aligned source-target pairs, can struggle when a single source acoustic pattern maps to multiple distinct target acoustic patterns. This ambiguity leads to "heavy" mixtures in the GMM, where multiple components contribute to the prediction, ultimately reducing the quality of the converted speech. The paper seeks to alleviate this problem by incorporating more informative context into the modeling process.

Methods

The paper builds upon the standard GMM-based VC framework. The core innovation lies in introducing context-dependent modeling, specifically by incorporating phonetic information into the GMM learning and classification stages.

  • Standard GMM-based VC: This involves aligning source and target speech frames, learning a GMM that models the joint distribution of source and target acoustic features, and then using this GMM to transform source features to target features. Classification (determining which GMM component best represents a source frame) is typically done using only the source acoustic information.
  • The "One-to-Many" Problem: The authors illustrate this by comparing classification using only source data versus using joint source-target data. They show that when a source frame maps to multiple target variations, source-only classification leads to less confident assignments (lower maximum mixture weights) and higher conversion error.
  • Context-Dependent Modeling (Phonetic GMM): The key proposed solution is to group GMM components based on phonetic labels rather than purely acoustic similarity. Instead of learning a single GMM for all speech, they learn a "Phonetic GMM" where each component is associated with a specific phoneme. During transformation, classification is performed by directly assigning the frame to the GMM component corresponding to its phonetic label, effectively eliminating the ambiguity of acoustic-based mixture weight calculation for the "one-to-many" problem.

Datasets

The authors used parallel corpora of French speakers from the France Telecom's TTS system, "Baratinoo." This dataset consists of a female speaker (source) and a male speaker (target). The speech was sampled at 16kHz and segmented into phones and diphones. For their analysis, they focused on voiced phonemes and used the middle frame of each phone along with its two adjacent frames (the "stable" part of the phone). Approximately 82,000 feature vectors were used, with half for learning and half for testing.

Key Insights and Ideas

  • The "One-to-Many" Effect is Real and Detrimental: The paper provides clear empirical evidence that the one-to-many mapping problem is a significant source of error in standard GMM-based VC. The difference in performance between source-only and joint classification directly highlights this issue.
  • Phonetic Context is Crucial: Simply relying on acoustic similarity for GMM component assignment is insufficient. Incorporating phonetic context allows the model to better disambiguate source frames, leading to more accurate mappings.
  • Stable vs. Transition Frames Matter: The analysis suggests that stable parts of phones are more amenable to direct conversion, while transition frames introduce additional challenges and may benefit from different processing strategies (e.g., interpolation).

Key Results

  • Significant MSE Improvement: Using the proposed context-dependent (Phonetic GMM with phoneme separation) approach, the Mean Squared Error (MSE) for spectral conversion improved by approximately 0.9dB compared to the traditional acoustically-based GMM with source decoding.
  • Isolation of One-to-Many Mappings: The authors developed a criterion to classify frames as "one-to-one" or "one-to-many." They found that for frames identified as "one-to-many," the MSE difference between source and joint decoding was substantial (over 1.5dB), confirming the impact of this problem.
  • Phoneme Separation Benefits One-to-Many Frames: When analyzing the Phonetic GMM with the one-to-many mapping criterion, phoneme separation improved the MSE by over 1.2dB specifically for one-to-many frames, while one-to-one frames showed minimal change. This demonstrates the targeted benefit of the proposed method.
  • Transition Frames Degrade Performance: Including transition frames in the analysis led to a general decrease in performance across all methods, indicating that stable phone parts are more critical for accurate conversion.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while the Phonetic GMM with phoneme separation offers significant gains, it sacrifices some of the optimal (but unrealizable) performance achieved by joint acoustic decoding.
  • Informal listening tests indicated that distinguishing between different decoding strategies was difficult due to degraded synthesized speech quality, suggesting that perceptual evaluation is still needed.
  • The analysis of transition frames suggests that the current approach might not be optimal for these segments.
  • Future Work:
    • The authors propose exploring alternative methods for aligning speaker feature vectors to better capture acoustic similarity.
  • They aim to incorporate more temporal and contextual information to create more acoustically homogeneous speaker spaces.
  • Investigating separate treatment of stable and transition frames, possibly using interpolation for transitions, is suggested.
  • The paper mentions that while their method improves parameter conversion, further work is needed to enhance the overall speech synthesis quality.

Summary updated

BibTeX
@inproceedings{godoy2009alleviating,
  author = {Elizabeth Godoy and Olivier Rosec and Thierry Chonavel},
  title = {{Alleviating the one-to-many mapping problem in voice conversion with context-dependent modeling}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure