Evaluation of speaker mimic technology for personalizing SGD voices

Esther Klabbers, Alexander Kain, Jan P. H. van SantenInterspeech 2010

TL;DR

This paper presents a novel approach to personalize voices for Speech Generating Devices (SGDs) by transforming high-quality synthetic voices to mimic a target speaker using a limited set of recordings. Their "Speaker Mimicry" (SM) system combines Voice Transformation (VT) for spectral characteristics and Prosody Mimicry (PM) for intonation and timing, achieving significant improvements in speaker identification compared to generic synthetic voices.

What Problem Does This Paper Address?

Individuals who cannot communicate verbally often rely on Speech Generating Devices (SGDs). While modern SGDs can produce synthetic speech, they typically lack the ability to generate a voice that sounds like the user's own past speech. Existing systems like ModelTalker require extensive recordings (around 1650 sentences, ~68 hours) from the user, which is often infeasible for individuals with limited speaking ability or consistency. This paper addresses the challenge of creating personalized synthetic voices for SGD users with minimal recording effort, aiming to improve user acceptance and communication effectiveness.

Methods

The core of this work lies in the "Speaker Mimicry" (SM) system, which leverages two key technologies:

  • Voice Transformation (VT): This technique modifies the spectral characteristics of a source speaker's voice to match those of a target speaker. The authors employ two VT algorithms:
  • VTmax: Uses Gaussian Mixture Models (GMMs) to learn a mapping between source and target speaker's Linear Prediction Coding (LPC) coefficients. This method is suitable for higher-quality recordings.
  • VTmin: A more robust method that works with lower-quality recordings by focusing on formant frequencies and performing piecewise linear warping in the frequency domain. This is particularly useful when dealing with less-than-ideal recordings or users with inconsistent pronunciation.
  • Prosody Mimicry (PM): This component adapts the prosodic features (intonation and timing) of the synthetic speech to match the target speaker. The authors use models derived from the Generalized Linear Alignment Model (GLAM) to capture detailed pitch contours and accent patterns, going beyond simple average speaking rate or pitch range. They also mention using a Sums-of-Products model for duration.

The SM system integrates these VT and PM techniques into a Text-to-Speech (TTS) engine.

Datasets

The paper describes the creation of several datasets:

  • Data Set 1 (Target Set): 20 phonetically rich sentences used as a reference for natural speech quality in the evaluation.
  • Data Set 2: 276 sentences containing 257 diphones and 19 triphones, necessary for building concatenative TTS inventories for the 20 test sentences.
  • Data Set 3 (Training Set): A set of 46 sentences selected from Switchboard conversations using a greedy algorithm to cover the diphones and triphones in Data Set 2. This dataset represents a casual speaking style.

For their proof-of-concept, they recorded these datasets from 12 healthy speakers (4 male adults, 4 female adults, 2 male children, 2 female children). Each speaker served as both a source and a target for VT, resulting in 342 sentences per speaker.

Key Insights and Ideas

  • Minimal Data is Key: The fundamental idea is that a small set of recordings (less than 50 sentences) from an SGD user can be sufficient to adapt a high-quality synthetic voice, making personalized voices accessible to a wider range of users.
  • Prosody is Crucial: The results strongly suggest that Prosody Mimicry (PM) plays a critical role in making synthetic voices sound like a target speaker, both on its own and in amplifying the effects of Voice Transformation (VT).
  • VTmin's Robustness: The VTmin algorithm shows promise for adapting voices even from lower-quality or less consistent recordings, which is highly relevant for real-world SGD users.
  • Canonical Speakers as Sources: The need to record several "canonical" speakers as sources is highlighted, implying that finding optimal source-target speaker combinations is important for achieving the best mimicry.

Key Results

A perception experiment involving 12 listeners was conducted to evaluate the effectiveness of the SM system. The results showed:

  • Significant Improvement: In most conditions, performance (speaker identification accuracy) was significantly above chance.
  • PM's Impact: Prosody Mimicry (PM) significantly improved performance compared to generic diphones and also enhanced the results of both VTmax and VTmin.
  • VTmax Performance: VTmax, when combined with personalized prosody, performed well.
  • VTmin Performance: VTmin, when combined with personalized prosody, also yielded significant improvements over generic diphones.
  • Qualitative Findings: The paper implies that the SM system can create synthetic voices that are perceived as more similar to the target speaker than generic synthetic voices.

Limitations and Future Work

  • Limitations:
    • The current study used studio-quality recordings. The ultimate goal is to adapt voices from arbitrary recordings (e.g., old videos), which may present greater challenges.
  • The current PM parameter estimation relies on manual inspection of pitch contours, which is not scalable.
  • The paper acknowledges that not every source voice is equally suitable for transformation to a given target voice, suggesting the need for careful selection of canonical source speakers.
  • Future Work:
    • Improve VT algorithms for more natural-sounding speech.
  • Implement a duration adaptation algorithm to better match target speaker durational patterns.
  • Develop automatic parameter estimation algorithms for pitch contours using GLAM.
  • Develop an SGD interface and gather feedback from actual SGD users.
  • Evaluate SM capability not only for speaker discriminability but also for intelligibility and naturalness.

Summary updated

BibTeX
@inproceedings{klabbers2010evaluation,
  author = {Esther Klabbers and Alexander Kain and Jan P. H. van Santen},
  title = {{Evaluation of speaker mimic technology for personalizing SGD voices}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure