Atom decomposition-based intonation modelling

Pierre-Edouard Honnet, Branislav Gerazov, Philip N. GarnerICASSP 2015 · OpenReview

Abstract

Current statistical parametric text-to-speech (TTS) synthesis methods allow production of neutral speech with acceptable quality. However, prosody is often qualified as unsatisfactory and sounding too flat. In this paper, we address intonation modelling for TTS based on physiological aspects of prosody production. A set of gamma distribution shaped atoms is defined and then intonation decomposition is performed using a matching pursuit algorithm. Some preliminary experiments show that this model allows easy extraction of physiologically meaningful atoms that could be used to generate intonation in a TTS system.

Introduction

Modern Text-to-Speech (TTS) systems often suffer from a "prosody gap." While they can pronounce words accurately, the resulting speech frequently sounds robotic or "flat" because it lacks natural intonation—the rise and fall of pitch that conveys emotion and emphasis. Traditional systems typically model pitch (fundamental frequency, $F_0$) frame-by-frame, ignoring the physical reality of how humans produce sound.

Human intonation is the result of specific muscle contractions in the larynx. This research proposes a mathematical model that mimics these physiological "twitches" by breaking down pitch contours into "atoms" based on the physics of human throat muscles. This approach aims to create more natural, less computer-like synthetic speech.

The Atom Decomposition Model

The core of the proposed method is Atom Decomposition, where any complex pitch curve is represented as a sum of simple, physiologically-plausible atoms.

  1. Physiological Atoms: Instead of arbitrary mathematical shapes, the model uses functions based on the general gamma distribution. This choice is motivated by previous research suggesting that laryngeal muscle responses to nerve impulses behave like "critically damped systems," which naturally produce gamma-distribution-like shapes.
  2. Two-Layer Representation: Intonation is split into a long-term phrase component (representing the sentence's overall shape) and short-term local components or atoms (representing accents or syllable-level emphasis).
  3. Matching Pursuit (MP): To analyze real speech, a Matching Pursuit algorithm is used. This "greedy" solver iteratively identifies the best-fitting atom, subtracts it from the signal, and repeats the process on the residual until the curve is accurately described.
Decomposition of a pitch contour into a phrase component and 9 local atoms.

Experimental Evaluation and Results

The model was validated across three languages—English, French, and German—using a dataset of 60 sentences from six different speakers. This multi-language approach tests the model's claim of being language-independent, as it relies on universal human physiology rather than language-specific linguistic rules.

Key Insights

  • Optimal System Order: While the classic Fujisaki model uses a 2nd-order system, this study found that 4th-order systems ($k=4$) provide a significantly better fit for human speech, suggesting higher physiological complexity than previously assumed.
  • Efficiency and Accuracy: The model achieves high accuracy with minimal complexity. Using approximately one atom per syllable allows the reconstructed pitch to be perceptually indistinguishable from the original recording.

Quantitative Performance

The model's accuracy was measured using weighted Root Mean Square Error (wRMSE) and weighted correlation (wCorr). As shown in the performance plots, the error drops sharply as the number of atoms per syllable increases, with correlation approaching 1.0.

Average wRMSE and wCorr across speakers as a function of the number of atoms per syllable.

Conclusion and Future Directions

The atom decomposition model successfully demonstrates how complex intonation can be analyzed into physiologically meaningful components. Informal listening tests confirm that speech resynthesized from these atoms sounds virtually identical to natural speech.

The primary challenge moving forward is the transition from analysis (decomposition) to synthesis (generation). Future work will focus on "dictionary learning" to refine atom sets for specific speakers and linking these atoms to semantic meanings. This would enable TTS systems to automatically place the correct atoms to produce expressive and controllable synthetic voices.

Summary updated

Abstract

Current statistical parametric text-to-speech (TTS) synthesis methods allow production of neutral speech with acceptable quality. However, prosody is often qualified as unsatisfactory and sounding too flat. In this paper, we address intonation modelling for TTS based on physiological aspects of prosody production. A set of gamma distribution shaped atoms is defined and then intonation decomposition is performed using a matching pursuit algorithm. Some preliminary experiments show that this model allows easy extraction of physiologically meaningful atoms that could be used to generate intonation in a TTS system.
BibTeX
@inproceedings{honnet2015atom,
  author = {Pierre-Edouard Honnet and Branislav Gerazov and Philip N. Garner},
  title = {{Atom decomposition-based intonation modelling}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2015},
  url = {https://openreview.net/forum?id=vibl1GPXqoS},
}
Figure