Speaker-independent HMM-based voice conversion using quantized fundamental frequency

Takashi Nose, Takao KobayashiInterspeech 2010

TL;DR

This paper introduces a novel voice conversion (VC) technique that significantly reduces the need for paired training data by leveraging speaker-independent models for decoding and speaker adaptation for synthesis. The method achieves competitive or superior performance compared to existing methods, even with as little as 10 adaptation sentences.

What Problem Does This Paper Address?

Voice conversion (VC) aims to change the speaker's identity in an audio utterance while preserving the linguistic content. Traditional methods, like GMM-based spectral mapping, often require large amounts of parallel (paired) speech data from both source and target speakers. This is a significant bottleneck, making it difficult to convert voices for new speakers without extensive recording sessions. Furthermore, these frame-based methods struggle to capture the dynamic, segmental characteristics of speaker individuality. While segment-based approaches offer improvements, they still typically demand substantial training data for both speakers. This paper tackles the challenge of reducing the data requirements for speaker-independent VC, specifically aiming to eliminate the need for source speaker training data and minimize target speaker training data.

Methods

The proposed approach builds upon HMM-based speech synthesis and recognition. The core innovation lies in two key areas:

  • Speaker-Independent (SI) Decoding: Instead of using a speaker-dependent model to decode the input speech into phonetic and prosodic symbols, the authors employ a speaker-independent model. This is feasible when the phonetic transcription is already known, simplifying the decoding task to phoneme segmentation. This effectively removes the requirement for source speaker training data.
  • Speaker Adaptation (SA) for Synthesis: For the synthesis part, the authors utilize speaker adaptation techniques, specifically adapting an "average voice model" (which they refer to as the SI model) to the target speaker. This technique, well-established in text-to-speech synthesis, is highly effective when only a limited amount of target speaker data is available.

The system first decodes the source speech into phonetic, mel-cepstral, and quantized fundamental frequency (Fo) sequences. The quantized Fo symbols capture prosodic information. These sequences are then used to generate a label sequence for synthesis. Finally, a speaker-adapted HMM is used to generate converted speech parameters, which are then synthesized into audio. A key detail is the use of quantized Fo symbols, derived from global mean and variance of log Fo within an utterance, to represent prosodic context.

Datasets

The experiments utilized the ATR Japanese speech database set B.

  • Source Speakers: One male (MSH) and one female (FYM).
  • Target Speaker: One female (FKN).
  • Training Data: Phonetically balanced 200 sentences per speaker (subsets A-D).
  • Test Data: 53 sentences (subset J).
  • SI Models: Trained on 450 sentences per speaker from six speakers not included in the source/target sets, totaling 2700 sentences.

Key Insights and Ideas

  • Eliminating Source Data is Viable: The most significant insight is that using a speaker-independent model for decoding is effective, provided phonetic transcriptions are available. This drastically simplifies the data acquisition process for VC.
  • Adaptation is Powerful for Limited Data: Speaker adaptation techniques, proven in TTS, are highly effective for VC, allowing for good conversion quality with very few target speaker sentences.
  • Quantized Fo Captures Prosody: Quantizing fundamental frequency provides a robust way to represent prosodic context for HMM-based synthesis, contributing to better conversion quality.
  • Segmental Information Matters: The segment-based approach, by considering phoneme-level dynamics, inherently captures more speaker individuality than frame-based methods.

Key Results

  • Reduced Data Requirements: The proposed HMM-based VC with SI decoding and SA synthesis achieved comparable or better results than GMM-based methods using 200 parallel sentences, with only 10 adaptation sentences for the target speaker.
  • Spectral Distortion: For male-to-female conversion, the proposed HMM-SA outperformed GMM200 with 10 adaptation sentences. For female-to-female, it outperformed GMM200 with 40 adaptation sentences. The proposed technique was also found to be less sensitive to source speaker variations.
  • Fo Distortion: The proposed technique outperformed linear transformation even with only 5 adaptation sentences.
  • Subjective Evaluation: The proposed technique with 10 adaptation sentences yielded better speaker similarity than GMM-based VC with 200 parallel sentences. Scores for 40 adaptation sentences were slightly better than speaker-dependent HMM with 200 sentences, possibly due to issues with unnatural phoneme duration in the SD model.

Limitations and Future Work

  • Phonetic Transcription Dependency: The effectiveness of the SI decoding relies on having accurate phonetic transcriptions for the input speech.
  • Future Work: The authors suggest exploring the technique with different types of speech, such as emotional or singing voices.

Summary updated

BibTeX
@inproceedings{nose2010speaker,
  author = {Takashi Nose and Takao Kobayashi},
  title = {{Speaker-independent HMM-based voice conversion using quantized fundamental frequency}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure