Modeling the creaky excitation for parametric speech synthesis

Thomas Drugman, John Kane, Christer GoblInterspeech 2012

TL;DR

This paper tackles the challenge of synthesizing natural-sounding creaky voice in speech synthesis systems. They introduce a novel extension to existing vocoder models that specifically captures the unique acoustic characteristics of creaky vocalizations, leading to significantly improved synthesis quality compared to traditional methods.

What Problem Does This Paper Address?

Modern speech synthesis systems, particularly those based on statistical parametric models (like HMMs), rely on accurately modeling the acoustic properties of speech sounds found in their training corpora. While significant progress has been made in modeling "modal" (normal) speech, many languages and conversational contexts frequently employ non-modal phonation, such as creaky voice. Creaky voice, characterized by long glottal pulse durations, minimal overlap between pulses, and secondary excitation peaks, poses a significant challenge for standard vocoders. These vocoders, designed for modal speech, fail to capture these distinctive features, resulting in unnatural or distorted synthesized creaky speech. This gap limits the naturalness and expressiveness of current text-to-speech (TTS) systems when dealing with corpora containing creaky voice, or when aiming for conversational or expressive speech synthesis.

Methods

The authors propose an extension to the Deterministic plus Stochastic Model (DSM) vocoder, specifically addressing the creaky excitation. Their approach involves a four-step analysis workflow:

  1. Creaky Voice Detection: Identifying segments of speech exhibiting creaky voice. While automated methods exist, the study relies on manual annotations for precision.
  2. GCI and Pitch Estimation: Accurately locating Glottal Closure Instants (GCIs) and estimating the fundamental frequency (pitch) in creaky voice. Standard GCI detectors often struggle with the altered glottal closing characteristics of creaky voice. The authors adapt a method using the SEDREAMS algorithm with a post-processing step involving a tuned resonator to filter out false alarms.
  3. Secondary Peak Estimation: Identifying the distinct secondary excitation peaks present in creaky speech. These are linked to glottal opening after a long closed period. The authors define an "open period" (time between a secondary pulse and the next GCI) and observe its remarkably consistent duration across speakers, even with varying pitch.
  4. Data-driven Waveform Extraction: This is the core of their novel model. The creaky excitation is modeled as having deterministic and stochastic components, similar to DSM. However, they split the analysis into "open" and "closed" periods. For each period, data-driven waveforms are extracted: the first eigenvector from Principal Component Analysis (PCA) for the deterministic component, and the energy envelope for the stochastic component. These components are then used to reconstruct the creaky excitation.

The proposed vocoder then integrates these extracted components. The deterministic component is derived from the first eigenvector, with the closed period duration being resampled to match the target pitch. The stochastic component uses white Gaussian noise modulated by the energy envelope, again with the closed period duration being resampled. The final residual signal is the sum of these components, and this is then filtered by an MLSA filter using Mel-Generalized Cepstral (MGC) coefficients to produce the creaky voice. For non-creaky segments, the standard DSM is used.

Datasets

The study utilizes two speech synthesis databases:

  • The BDL voice (male American speaker) from the CMU ARCTIC database.
  • The MV database (Finnish male speaker).

Both databases were downsampled to 16 kHz. Approximately 100 sentences containing creaky regions were selected from each database for analysis and annotation.

Key Insights and Ideas

  • Creaky Voice is More Than Just Low Pitch: The study highlights that creaky voice has distinct acoustic characteristics beyond just a low fundamental frequency, particularly in the excitation signal.
  • The "Open Period" is Key: The consistent duration of the "open period" (time between secondary excitation and GCI) across speakers, even with pitch variations, is a crucial observation. This consistency simplifies the modeling process.
  • Data-Driven Components Capture Nuance: Using PCA eigenvectors and energy envelopes derived from actual creaky speech data allows the model to capture the specific timbral qualities of creaky voice, which rule-based or simpler models miss.
  • Standard Vocoders Fall Short: The research empirically demonstrates that traditional vocoders, designed for modal speech, are inadequate for synthesizing creaky voice, leading to a clear perceptual difference.

Key Results

Subjective listening tests were conducted to evaluate the proposed vocoder against the traditional DSM vocoder:

  • ABX Test: Approximately 60% of responses indicated that the proposed model produced speech closer to the original creaky segments, compared to 15% for the traditional approach.
  • CMOS Test: The proposed model was preferred over the traditional vocoder in 55% of cases. Specifically, 20-30% of listeners rated the proposed model as "better" or "much better," while the traditional model was rated as "better" or "much better" in less than 1% of cases. The averaged CMOS scores were 0.71 ± 0.14 for BDL and 0.88 ± 0.16 for MV, confirming the superiority of the proposed model.

Limitations and Future Work

  • Manual Annotation Reliance: The study relies on manual annotation for creaky voice detection and GCI/secondary peak estimation. While this ensures accuracy, it's labor-intensive. Future work could focus on improving the robustness of automated detection methods.
  • Limited Speaker Range: The analysis was performed on two male speakers. While the findings suggest a general pattern, further research with a wider range of speakers (including females) and different types of creaky voice would be beneficial.
  • Generalization to Other Non-modal Voices: The proposed model is specific to creaky voice. Extending this framework to model other non-modal phonation types (e.g., breathy voice, vocal fry variations) is a natural next step.
  • Perceptual Variability: While the results show a clear preference for the proposed model, a significant portion of responses indicated equivalence or no strong preference. Further investigation into the factors contributing to this variability could refine the model.

Summary updated

BibTeX
@inproceedings{drugman2012modeling,
  author = {Thomas Drugman and John Kane and Christer Gobl},
  title = {{Modeling the creaky excitation for parametric speech synthesis}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure