Cross-lingual speaker adaptation for HMM-based speech synthesis based on perceptual characteristics and speaker interpolation

Viviane de Franca Oliveira, Sayaka Shiota, Yoshihiko Nankaku, Keiichi TokudaInterspeech 2012

TL;DR

This paper introduces a novel cross-lingual speaker adaptation (CLSA) method for HMM-based speech synthesis. By leveraging a language-independent "perceptual characteristics" (PC) space, it allows speech synthesis systems to mimic a target speaker's voice without requiring bilingual data, a significant hurdle in previous approaches. The method shows promise in improving naturalness and voice similarity compared to existing techniques.

What Problem Does This Paper Address?

Speech-to-speech translation (S2ST) systems aim to translate not only the content but also the voice characteristics of the original speaker. Cross-lingual speaker adaptation (CLSA) is crucial for this, enabling synthesized speech in the target language to sound like the source speaker. Existing HMM-based CLSA methods often rely on state mapping (SM) that extracts speaker characteristics directly from acoustic features. However, these transforms can retain language-specific dependencies, impacting the synthesized speech quality in the target language. Furthermore, adapting prosodic information, which is highly language-dependent, remains a significant challenge. Building bilingual databases for training is also expensive and difficult to scale. This paper addresses the need for a CLSA method that is language-independent, avoids the need for bilingual data, and can potentially adapt prosodic features.

Methods

The proposed method introduces a language-independent "perceptual characteristics" (PC) space as an intermediary for mapping between source and target languages. Here's a breakdown:

  1. Perceptual Characteristics (PCs) Extraction: Subjective listening tests are conducted where volunteers rate speech samples based on a set of contrasting perceptual characteristics (e.g., gloomy/bright, low-pitched/high-pitched). These ratings are then analyzed using Principal Component Analysis (PCA) to create a language-independent PC space.
  2. Mapping to PC Space: For a new source speaker, their voice characteristics are expressed as a linear combination of existing speakers in the source language space. The weights from this combination are then used to find a corresponding point in the PC space.
  3. Speaker Interpolation: This PC space point is then used to estimate weights for interpolating speaker models in the target language space. Crucially, the final speech synthesis uses only the acoustic models of the target language, thus avoiding source language acoustic influence.
  4. Prosodic Adaptation: The language-independent nature of the PC space offers the potential to adapt prosodic information (like pitch and duration) from the target speaker, which is difficult with purely acoustic-based methods.

This approach bypasses the need for bilingual data by using the PC space as a universal representation of speaker traits.

Datasets

  • WSJO database: Used for English speech samples (source language).
  • Japanese Newspaper Article Sentences (JNAS) database: Used for Japanese speech samples (target language).

The paper describes a listening test conducted with a total of 28 speakers (male and female, from both Japanese and English) to extract perceptual characteristics. For experiments, 20 female speakers from WSJO and JNAS were used for training, and 50 sentences were used for adaptation and testing.

Key Insights and Ideas

  • Language-Independent Representation is Key: The core idea is that certain perceptual voice characteristics are universal and not tied to language. By mapping to this abstract space, language dependencies can be effectively bypassed.
  • Bypassing Bilingual Data: This is a major practical advantage. Creating large, high-quality bilingual speech datasets is a significant bottleneck in CLSA research. This method offers a way around it.
  • Potential for Prosodic Adaptation: The language-independent PC space opens the door to adapting prosody, a crucial aspect of natural speech that has been challenging for previous methods.
  • Synthesized Speech Quality: By using target language acoustic models exclusively for synthesis, the method avoids the negative influence of source language acoustics, leading to more natural-sounding output.

Key Results

  • Subjective Evaluation:
    • Naturalness (MOS test): The proposed method (PC-1 and PC-2) achieved higher naturalness scores than the no-adaptation baseline and the SM-based method. This indicates that the synthesized speech sounds more natural.
  • Voice Similarity (DMOS test): The proposed method (PC-1) performed comparably to the SM-based method, and better than no-adaptation. However, a more advanced version (PC-2) that performed separate interpolation for spectrum, F0, and duration did not outperform PC-1.

Limitations and Future Work

  • Speaker Interpolation Performance (PC-2): The speaker interpolation approach (PC-2), which attempted to interpolate spectrum, F0, and duration separately, did not outperform the simpler PC-1 approach. The authors attribute this to:
  • Sparsity of training speakers: The current training set might not be large enough to adequately cover the PC space.
  • Model mismatch: Combining separately interpolated speaker models might lead to mismatches.
  • Future Work:
    • Investigate the optimal number of training speakers needed to cover the PC space.
  • Explore new methods for estimating weights and performing speaker interpolation.
  • Examine different ways of selecting perceptual characteristics and constructing the PC space.
  • Conduct tests with "open speakers" (speakers not in the training set).

Summary updated

BibTeX
@inproceedings{oliveira2012cross,
  author = {Viviane de Franca Oliveira and Sayaka Shiota and Yoshihiko Nankaku and Keiichi Tokuda},
  title = {{Cross-lingual speaker adaptation for HMM-based speech synthesis based on perceptual characteristics and speaker interpolation}},
  booktitle = {Interspeech},
  year = {2012},
}

Loading related content...

Figure