A KL Divergence and DNN-Based Approach to Voice Conversion without Parallel Training Sentences

Feng-Long Xie, Frank K. Soong, Haifeng LiInterspeech 2016

TL;DR

This paper introduces a novel voice conversion (VC) method that eliminates the need for parallel speech data between source and target speakers. By leveraging a Speaker-Independent Deep Neural Network (SI-DNN) for phonetic representation and Kullback-Leibler Divergence (KLD) for mapping, the approach achieves high speaker similarity and spectral accuracy, outperforming existing non-parallel VC methods.

What Problem Does This Paper Address?

Traditional voice conversion (VC) systems, which aim to change a speaker's voice while preserving the linguistic content, have historically relied heavily on parallel training data. This means having recordings of the exact same sentences spoken by both the source and target speakers. Acquiring such parallel data is a significant bottleneck, requiring extensive and costly data collection and alignment efforts. This paper tackles the challenge of performing effective voice conversion without this parallel data requirement, making VC more accessible and practical.

Methods

The core of the proposed approach is the "KLD-DNN" method, which combines two key components:

  • Speaker-Independent DNN (SI-DNN) for Phonetic Representation: The authors train a SI-DNN, typically used for Automatic Speech Recognition (ASR), on a large dataset of diverse speakers. This DNN learns to map acoustic speech features to "senones" (phonetic states) in a speaker-independent manner. Crucially, the output of this DNN is a posterior probability distribution over these senones for each speech frame. This distribution effectively captures the phonetic content of the speech, abstracting away speaker-specific characteristics.
  • Kullback-Leibler Divergence (KLD) for Mapping: KLD is used as a measure of "distance" or divergence between probability distributions.
  • Supervised Mode: When target speaker transcriptions are available, the SI-DNN's output posteriors for the source speaker are mapped to the senones of a target speaker's GMM-HMM Text-to-Speech (TTS) system. The mapping is achieved by finding the target senone whose posterior distribution (learned from the target TTS) has the minimum KLD with the source frame's posterior distribution.
  • Unsupervised Mode: When target speaker transcriptions are not available, the approach first clusters the target speaker's phonetic representations (derived from the SI-DNN) into phonetic clusters. Then, each source speaker's frame is mapped to the closest phonetic cluster centroid using KLD.

Finally, a maximum probability trajectory generation algorithm is used to synthesize the converted speech. The paper also briefly mentions an optional "Speaker Dependent DNN" (SD-DNN) adaptation for potentially improved performance when speaker-specific data is available.

Datasets

  • Training:
    • Target Speaker TTS Training: GMM-HMM TTS models for the target speaker were trained using their speech and transcriptions.
  • SI-DNN Training: A Speaker-Independent DNN (SI-DNN) was trained on the Wall Street Journal CSR corpus, which contains 78 hours of speech from 284 native American English speakers (SI-284 dataset).
  • Evaluation:
    • CMU ARCTIC database: Used for voice conversion experiments. Specifically, recordings from speakers BDL (US Male), CLB (US Female), and RMS (US Male) were used.
  • Test Sets: 50 utterances were used as test sets for each conversion pair.

Key Insights and Ideas

  • Phonetic Abstraction via SI-DNN: The SI-DNN acts as a powerful feature extractor, converting raw acoustic features into a speaker-independent phonetic representation (posterior probabilities). This is key to decoupling the linguistic content from speaker identity.
  • KLD as a Robust Mapping Metric: KLD proves to be an effective metric for measuring the similarity between phonetic representations, allowing for accurate mapping even without direct frame-to-frame alignment.
  • Unsupervised VC is Achievable and Effective: The unsupervised mode, which doesn't require target speaker transcriptions, is shown to perform well, significantly broadening the applicability of the method.
  • Phonetic Clusters Outperform TTS Senones in Unsupervised Mode: The paper suggests that clustering phonetic representations (DNN posteriors) is more consistent with the KLD mapping process than using pre-defined TTS senones, leading to better results in the unsupervised scenario.

Key Results

  • Competitive Performance: Both the senone mapping (TSM) and phonetic cluster mapping (PCM) approaches achieved performance close to the "upper bound" of a TTS system trained directly on the target speaker's data.
  • PCM Superiority in Unsupervised Mode: In unsupervised voice conversion, the phonetic cluster mapping (PCM) method achieved better results (lower Log Spectral Distortion - LSD) than senone mapping (TSM). For example, for SLT to BDL, PCM achieved an LSD of 4.56 dB compared to TSM's 4.68 dB.
  • Data Efficiency: The phonetic cluster mapping method was found to be data-efficient, with 100 utterances of the target speaker being adequate for constructing a good phonetic space. Even with as few as 10 utterances, the system achieved "fairly good intelligibility and decent similarity."
  • Outperformance of Baseline: The proposed KLD-DNN approach significantly outperformed a previously proposed Sequence Error Minimization (SEM-NN) baseline, both objectively (lower LSD) and subjectively. For instance, in the SLT to BDL conversion, KLD-DNN (PCM-A-100) achieved an LSD of 4.40 dB, while SEM-NN-100 had an LSD of 4.98 dB. Subjective listening tests also showed overwhelming preference for the KLD-DNN approach in terms of naturalness and similarity.

Limitations and Future Work

  • Prosody Transformation: The current study focuses on spectral characteristics and mentions that prosody is transformed on a global scale using a Gaussian normalized transformation. This suggests that fine-grained prosody control is not a primary focus and could be an area for future work.
  • Speaker Dependent DNN Adaptation: While mentioned as a potential improvement, the paper doesn't deeply explore the full benefits or optimal ways to integrate SD-DNN adaptation with the KLD-DNN framework.
  • Further Exploration of Unsupervised Clustering: While effective, the paper could explore more advanced clustering techniques or unsupervised adaptation strategies for even greater robustness.

Summary updated

BibTeX
@inproceedings{xie2016kl,
  author = {Feng-Long Xie and Frank K. Soong and Haifeng Li},
  title = {{A KL Divergence and DNN-Based Approach to Voice Conversion without Parallel Training Sentences}},
  booktitle = {Interspeech},
  year = {2016},
}
Figure