A model of the regularities underlying speaker variation: evidence from hybrid synthesis

Susan R. HertzInterspeech 2006

TL;DR

This paper proposes a "hybrid model" of speech perception that explains how listeners can understand speech despite massive variations between speakers. By demonstrating that consonant segments can be swapped between speakers with minimal impact on intelligibility, the authors provide strong evidence that listeners abstract away from speaker-specific acoustic details for these segments.

What Problem Does This Paper Address?

The human voice is incredibly diverse. Even the same person saying the same word can produce acoustically different sounds depending on context, emotion, or simply natural variation. When different people speak, these differences are amplified due to variations in vocal tract size, shape, and speaking style. This variability poses a significant challenge for speech recognition systems and for understanding how humans achieve robust speech perception. How do listeners consistently identify phonemes (the basic units of sound in a language) when the acoustic signal is so inconsistent? Some theories suggest listeners rely on vast memory stores of "exemplars" of speech sounds, while others propose more rule-based processing. This paper investigates a middle ground, proposing a model that distinguishes between more generic consonant segments and more speaker-identity-cued vowel segments.

Methods

The core of the paper's methodology lies in "hybrid synthesis." This technique involves creating speech by splicing together segments from different speakers. The authors developed a "hybrid model" that posits speech is processed into two main units: acoustic consonant clusters (ACCs) and acoustic nuclei (ANs). ACCs are thought to be more generic and less speaker-dependent, while ANs (primarily vowels) are more tied to speaker identity.

The experiments involved:

  • Hybrid Synthesis: Creating new speech utterances by replacing consonant segments from one speaker with those from another (or even synthetic speech).
  • Speaker Identification (SI) Experiments: Presenting listeners with these hybrid stimuli and asking them to identify the speaker or, if it didn't sound like a target speaker, to identify its gender and age.
  • Speech Quality (SQ) Experiments: Asking listeners to rate the naturalness of the hybrid stimuli and identify any perceived problems (e.g., unnaturalness, foreign accent).

The authors also employed rule-based and model-based formant synthesis to create synthetic segments for substitution.

Datasets

The paper does not explicitly list or introduce a formal dataset in the traditional machine learning sense. Instead, it utilizes:

  • Recordings from 11 "hybrid speakers": This group included 8 human speakers (3 children, 2 young adults, 3 older adults) and 3 synthetic speakers.
  • Target Speakers: Six of the human speakers served as "target speakers" whose voices were mimicked. These speakers recorded an 82-word passage for listener training.
  • Experimental Utterances: Four specific sentences were used for the core experiments, chosen for their inclusion of varied consonant clusters, unstressed syllables, and syllable-final consonants.

The paper focuses on the creation of novel stimuli through hybrid synthesis for evaluation, rather than using pre-existing, labeled datasets for training.

Key Insights and Ideas

  • Consonants are Surprisingly Robust: A major finding is that over half of the speech segments (60-70% of utterance duration) can be replaced by segments from different speakers (even opposite genders, different ages) with little perceptual impact. This suggests that consonant clusters are processed in a way that abstracts away from fine-grained speaker-specific acoustic details.
  • The "Hybrid Model" Distinction: The paper strongly supports its proposed "hybrid model," which differentiates between acoustic consonant clusters (ACCs) and acoustic nuclei (ANs). ACCs are shown to be highly substitutable, while ANs are implicitly linked to speaker identity.
  • Timing is Crucial for Consonants: The model emphasizes the importance of timing relations within and across ACCs for phoneme decomposition. These durational cues are presented as more stable and less affected by vocal tract variations than spectral cues.
  • Abstraction from Acoustic Differences: Listeners appear to "abstract away" from acoustic differences in ACCs that result from individual vocal physiologies, focusing instead on more general phonetic and phonological patterns.
  • Synthetic Segments Can Work: Even formant-synthesized consonant segments, produced with general rules, can serve as effective surrogates, indicating that rule-based processing can capture the necessary information for robust perception.

Key Results

  • Speaker Identification: Listeners familiar with the hybrid speakers correctly identified them 96% of the time, with an average similarity score of 1.99 (on a scale where 1 is "exactly like"). This was comparable to natural speech (98% correct, 1.37 score). Unfamiliar listeners also showed similar performance for hybrid and natural stimuli (78%/2.17 vs. 79%/1.97).
  • Speech Quality: Core hybrid stimuli received an average naturalness score of 1.97 (on a scale where 1 is "very natural"). This was significantly better than copy-based formant synthesis (4.21) and rule-based formant synthesis (4.84), but slightly less natural than corpus-based concatenation (CBWC1: 2.85).
  • Synthetic Consonants' Generality: Consonant surrogates from rule-based synthesis did not degrade speaker identification, even when the synthesis itself was judged less accurately.

Limitations and Future Work

  • Focus on English: The study primarily focuses on English. While preliminary results suggest the model extends to other languages, further cross-linguistic investigation is needed.
  • Implicit Role of ANs: While the paper posits ANs are key to speaker identity, the experimental evidence presented focuses more on the substitutability of ACCs. More direct experimental validation of the AN's role in speaker identity would be beneficial.
  • Definition of "Mimetic": The paper mentions synthetic segments being "mimetic" but doesn't deeply explore the nuances of what constitutes mimetic synthesis beyond intelligibility and naturalness.
  • Future Work: The authors suggest the model's applicability to other languages and imply further exploration of the AN's role in speech perception and speaker identity. The success of hybrid synthesis itself opens avenues for more sophisticated text-to-speech systems.

Summary updated

BibTeX
@inproceedings{hertz2006model,
  author = {Susan R. Hertz},
  title = {{A model of the regularities underlying speaker variation: evidence from hybrid synthesis}},
  booktitle = {Interspeech},
  year = {2006},
}
Figure