Synthetic F0 can effectively convey speaker ID in delexicalized speech

Eric Morley, Esther Klabbers, Jan P. H. van Santen, Alexander Kain, Seyed Hamidreza MohammadiInterspeech 2012

TL;DR

This paper explores whether fundamental frequency (F0) alone can convey speaker identity, even without spectral or segmental information. They developed two novel F0 synthesis methods based on the Linear Alignment Model (LAM) and found that synthetic F0 can convey speaker identity nearly as effectively as natural F0 in delexicalized speech, particularly for female speakers.

What Problem Does This Paper Address?

Identifying speakers is a complex task that humans perform effortlessly, often using a combination of vocal characteristics. While automated systems leverage various acoustic features for speaker identification, the specific contribution of F0 (pitch) to human speaker identification, especially in synthesized speech, remains underexplored. This paper aims to quantify how much speaker identity information is encoded in F0 and to develop methods for synthesizing F0 contours that effectively capture and convey this identity. The challenge lies in creating synthetic F0 that sounds natural and speaker-specific, even when stripped of other vocal cues.

Methods

The authors propose two F0 synthesis methods rooted in the Linear Alignment Model (LAM), a framework that decomposes F0 contours into phrase curves (overall trajectory) and accent curves (rise and fall associated with accented syllables).

  1. Parametric Synthesis: This method uses speaker-specific parameters estimated from decomposed F0 contours. It models the phrase curve with a piecewise linear function and uses a time-warped cosine template for accent curves. The alignment of accent curves is speaker-independent, simplifying the process.
  2. Recombinant Synthesis: This approach synthesizes F0 by recombining pre-existing component curves (phrase and accent) from a corpus. It requires detailed feature information about the target utterance (e.g., sentence type, phrasal position) to select and scale appropriate curves.

Both methods rely on an automated F0 decomposition algorithm that uses local minima to estimate the phrase curve and then subtracts it from the original F0 to obtain accent curves.

Datasets

The primary dataset used is the CSLU Fo Protocol (FP). This is a set of all-sonorant utterances designed to elicit F0 contours with controlled linguistic and prosodic features, including pause locations, pitch accent, and emphasis. The FP was collected from multiple adult speakers (both male and female, native English speakers). For this study, they used an abridged version of the FP and modeled the F0 contours of six speakers (three male, three female).

Key Insights and Ideas

  • F0 is a powerful speaker identifier: Even when all other vocal cues (spectral, segmental, durational) are removed, F0 alone carries significant information about speaker identity for human listeners.
  • Synthetic F0 can mimic natural F0: The proposed LAM-based synthesis methods can generate F0 contours that are remarkably effective at conveying speaker identity, approaching the effectiveness of natural F0.
  • Speaker-specific models are crucial: Models trained on individual speakers (PARAM-SPK and RECOMB) significantly outperform speaker-independent models (PARAM-GEN) in conveying speaker identity.
  • Gender differences exist: The effectiveness of synthetic F0 in conveying speaker identity appears to be more pronounced for female speakers compared to male speakers.

Key Results

A perceptual experiment was conducted where listeners identified speakers based on delexicalized speech stimuli.

  • Overall Effectiveness: Synthetic F0 from both speaker-specific methods (PARAM-SPK and RECOMB) conveyed speaker identity significantly better than a speaker-independent method (PARAM-GEN).
  • Comparison to Natural F0:
    • For female speakers, both PARAM-SPK and RECOMB conveyed speaker ID as effectively as natural delexicalized speech (NAT).
  • For male speakers, both speaker-specific methods conveyed speaker ID better than PARAM-GEN, but not quite as effectively as natural delexicalized speech (NAT). However, post-hoc analysis suggested that they might be as effective as natural speech.
  • Quantitative Results: Table 4 shows the proportion of correct responses. For example, for female speakers, NAT achieved 0.68 correct responses, while RECOMB achieved 0.69 and PARAM-SPK achieved 0.67. For male speakers, NAT achieved 0.60, RECOMB 0.55, and PARAM-SPK 0.55. Table 5 and 6 provide statistical significance values for these comparisons.

Limitations and Future Work

  • Interaction with other cues: The study focused solely on delexicalized speech, meaning interactions between F0 and other vocal cues (like spectral or durational information) that might carry speaker identity were not explored. These interactions could be important.
  • Sufficiency of mean and variance: The paper questions whether mean and variance of F0 are sufficient for speaker identity. Future work could involve manipulating LAM parameters to match mean and variance and re-evaluating.
  • AC alignment: The recombinant synthesis method uses a simple linear rescaling of ACs, which might not fully capture complex interactions like Fo peak alignment. Exploring different rescaling methods is suggested.
  • Generalizability: The study used a specific protocol (CSLU FP) and a limited number of speakers. Further validation with more diverse datasets and speakers would be beneficial.

Summary updated

BibTeX
@inproceedings{morley2012synthetic,
  author = {Eric Morley and Esther Klabbers and Jan P. H. van Santen and Alexander Kain and Seyed Hamidreza Mohammadi},
  title = {{Synthetic F0 can effectively convey speaker ID in delexicalized speech}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure