Synthesis of Expressive Speaking Styles with Limited Training Data in a Multi-Speaker, Prosody-Controllable Sequence-to-Sequence Architecture

Slava Shechtman, Raul Castro Fernandez, A. S. Sorin, David Haws2021 · DOI

TL;DR

This paper introduces a novel sequence-to-sequence speech synthesis architecture that can generate expressive speech with limited training data and transfer styles across speakers. It achieves this by incorporating controllable prosodic features, allowing for tunable expressiveness and a better trade-off between naturalness and style.

What Problem Does This Paper Address?

Generating expressive speech (e.g., happy, sad, angry) has been a long-standing challenge in Text-to-Speech (TTS) systems. While modern sequence-to-sequence (S2S) models like Tacotron have achieved impressive naturalness, they typically require large amounts of training data, which is a significant bottleneck when aiming for diverse expressive styles. Furthermore, transferring learned expressive styles to new speakers for whom no expressive data exists (cross-speaker style transplantation) is particularly difficult. Existing methods often struggle with disentangling style from text, require parallel data, or lack fine-grained control over the expressiveness level. This paper tackles the problem of creating expressive speech synthesis systems that are data-efficient, controllable, and capable of cross-speaker style transfer.

Methods

The core of the proposed system is a Tacotron2-like S2S architecture augmented with Hierarchical Prosodic Controls. These controls are designed to be speaker-agnostic and capture perceptually interpretable prosodic features at both sentence and word levels. These features include:

  • Durations: Average per-phone durations (Sdur, Wdur).
  • Pitch Dynamics: Range of pitch variations (Sfo, Wfo).
  • Pitch Contour: Slope of pitch changes (S∠fo, W∠fo).
  • Median Pitch: Average pitch level (Sfo, Wfo).

These eight prosodic features form a target vector that guides the synthesis process. Crucially, the system learns these prosodic controls in a fully supervised manner, avoiding the drawbacks of unsupervised latent style representations.

A key innovation is the controllable trade-off between expressiveness and naturalness. This is achieved through run-time additive offsets applied to the predicted prosodic controls. These offsets, tuned on a small development set, allow users to fine-tune the strength of a particular expressive style. This approach is contrasted with a "Classic Supervision" baseline that lacks these prosodic controls.

Datasets

The training material comprised four proprietary corpora from three professional native speakers of US English. This included:

  • A large corpus (9.5 hours) from a female speaker (F1).
  • Smaller expressive corpora (1 hour each for Good News - GDN, and Apologetic - APO styles) from the same female speaker (Flexp).
  • Two additional corpora from a female (F2) and a male (M1) speaker.

The Flexp corpus, containing labeled expressive data, was significantly smaller than the base corpora. For training, the Flexp data was replicated five times to compensate for its limited size. For evaluation, a set of 40 unseen sentences were used, with a neutral version of the proposed system serving as a quality reference.

Key Insights and Ideas

  • Supervised Prosodic Controls are Key: Directly learning and controlling perceptually meaningful prosodic features, rather than relying on unsupervised latent representations, leads to better disentanglement of style and controllability.
  • Tunable Expressiveness is Achievable: The use of run-time offsets on prosodic controls provides a simple yet effective mechanism to adjust the intensity of an expressive style, offering a desirable trade-off between expressiveness and naturalness.
  • Cross-Speaker Transplantation is Feasible: The speaker-agnostic nature of the prosodic controls, combined with the supervised learning approach, enables the system to transfer expressive styles to speakers for whom no labeled expressive data is available.
  • A Graceful Degradation: Increasing expressiveness leads to a slight, but acceptable, drop in overall speech quality (around 0.12-0.16 MOS), suggesting a sensible operating point for applications balancing expressiveness and naturalness.

Key Results

  • Generalizability: The proposed system significantly outperformed a classical supervised baseline in its ability to generalize to a target speaker lacking labeled expressive data (transplant condition).
  • Expressiveness Preference: Listeners strongly preferred the proposed system's ability to convey expressiveness over the baseline.
  • Trade-off: The system offers a reasonable trade-off between expressiveness and naturalness, allowing users to tune the level of expression.
  • MOS Scores: Mean Opinion Scores (MOS) for overall naturalness and quality showed that the expressive systems (PCExp, PCExp+) had slightly lower scores than the neutral system (PCNeu) and the baseline (BaseExp), but the degradation was minimal (e.g., 3.82 vs. 4.00 for speaker F1 in the moderate expressive condition).

Limitations and Future Work

  • Limited Data for Style: While the system excels with limited expressive data, the base corpora still consist of proprietary data from a limited number of speakers.
  • Prosody Focus: The current work primarily focuses on prosodic features. Future work aims to incorporate other style-modulating factors like timbre and voice quality, which are crucial for conveying a wider range of affects.
  • Quality-Expressivity Trade-off: Further research is needed to improve the balance between speech quality and expressiveness.
  • Spectral Characteristics: While the system can reproduce spectral characteristics consistent with pitch (e.g., lower pitch correlating with lower spectral energy), explicitly modeling timbre and voice quality would enhance expressiveness.

Summary updated

BibTeX
@misc{shechtman2021synthesis,
  author = {Slava Shechtman and Raul Castro Fernandez and A. S. Sorin and David Haws},
  title = {{Synthesis of Expressive Speaking Styles with Limited Training Data in a Multi-Speaker, Prosody-Controllable Sequence-to-Sequence Architecture}},
  year = {2021},
  doi = {10.21437/interspeech.2021-1446},
}
Figure