Weakly-Supervised Phrase Assignment from Text in a Speech-Synthesis System Using Noisy Labels

Asaf Rendel, Raul Castro Fernandez, Zvi Kons, Andrew Rosenberg, Ron Hoory, Bhuvana Ramabhadran2017 · DOI

TL;DR

This paper tackles the challenge of teaching Text-to-Speech (TTS) systems to correctly segment spoken sentences into meaningful phrases, a crucial step for naturalness. They achieve this by developing a novel "weakly-supervised" method that leverages existing, imperfect data sources to train a phrase assignment model, significantly improving perceived speech quality without requiring extensive manual annotation.

What Problem Does This Paper Address?

Generating natural-sounding synthetic speech requires more than just pronouncing words correctly; it involves conveying the intended meaning and flow through prosody – the rhythm, stress, and intonation of speech. A key component of prosody is intonational phrasing, where sentences are broken down into smaller, meaningful units. Properly segmenting input text into these phrases is a fundamental task for the "front-end" of a TTS system.

The challenge lies in obtaining high-quality data for training models that can perform this segmentation. Manually annotating text for phrase boundaries is time-consuming and expensive. Existing automatic methods often rely on linguistic rules or require perfectly aligned phonetic data, which might not be available or might not match the target domain of the TTS system. This paper aims to bridge this gap by developing a method that can learn phrase assignment from "noisy" or imperfect labels, reducing the dependency on meticulously hand-annotated datasets.

Methods

The core of the paper's contribution is a weakly-supervised phrase assignment model built using a bidirectional recurrent neural network (BiRNN). Instead of relying on perfect ground truth, they construct a "noisy" ground truth by merging heuristics from several existing resources:

  • Automatic high-quality phonetic alignments: These provide information about speech sounds and their durations.
  • Linguistic rules: Pre-existing rules that define syntactic structures and potential phrase breaks.
  • A legacy acoustic phrase-labeling system: An older system that already attempts to label phrase breaks.

This combined approach creates a dataset where labels might not be perfectly accurate but are sufficient for training.

The BiRNN model takes various features as input, including:

  • Punctuation type: The presence and type of punctuation marks.
  • Unigram log probability: The likelihood of a word appearing in isolation.
  • Normalized Pairwise Mutual Information (NPMI): Measures how often words co-occur with their neighbors.
  • Word embeddings: Learned representations of words that capture semantic relationships.

Crucially, they also employ a feature learning strategy. They train a separate model (a "comma predictor") on a large, unlabeled corpus (Gigaword) to predict punctuation. The internal activations of this model are then used as learned features for the main phrase assignment task. This allows the model to leverage information from a much larger text corpus without explicit phrase annotations.

The model is trained in a multi-task framework, predicting not only phrase boundaries but also auxiliary targets like normalized silence duration and whether a word is part of a short multi-adjective sequence. This helps the model learn more nuanced aspects of prosody.

Datasets

  • Training Data: A phonetically-aligned corpus of approximately 150,000 words, recorded by a female professional native speaker of North American English. This corpus was used to derive the "noisy" ground truth labels.
  • Gigaword Corpus: Used for feature learning (training the "comma predictor"). This is a large, unlabeled text corpus.
  • Evaluation Data:
    • The same 150K-word corpus used for training.
  • The Boston Radio University Corpus (BRUC) for evaluating generalization to a different domain.
  • Test sentences from three domains (general news, expert evidence, study evidence) for subjective evaluation.

The paper does not introduce a new benchmark dataset but rather leverages existing resources to create a novel training methodology.

Key Insights and Ideas

  • "Noisy" labels are sufficient: The paper demonstrates that even imperfect, heuristically derived labels can be effectively used to train a powerful phrase assignment model, significantly reducing the need for expensive manual annotation.
  • Feature learning from related tasks: Using a "comma predictor" model trained on a large unlabeled corpus to generate features for the phrase assignment task is a powerful way to leverage vast amounts of text data.
  • Multi-task learning benefits: Training the BiRNN to predict multiple related targets (phrase breaks, silence duration, adjective sequences) leads to a more robust and capable model.
  • Bridging the gap between syntax and prosody: The model effectively learns to map textual and acoustic cues to prosodic phrasing, a critical step for natural speech synthesis.

Key Results

  • Objective Evaluation: The proposed model achieved an overall F1-measure of around 90% for phrase assignment, significantly outperforming a baseline system that relies solely on linguistic rules.
  • Subjective Evaluation (Perceptual Listening Tests):
    • For both a female (FV) and a male (MV) synthetic voice, listeners rated the phrasing quality of speech generated with the proposed model significantly higher than that of the baseline system (p-values of .0031 and .002 respectively).
  • Crucially, this improvement in phrasing quality did not degrade overall naturalness (p-values of .154 and .0071 respectively).
  • The benefits were observed across different genres of text, though the specific genres that showed improvement varied between the two voices.

Limitations and Future Work

  • Dependency on existing heuristics: While reducing the need for hand-labeling, the method still relies on existing linguistic rules and an acoustic labeler (AuToBi) for generating the "noisy" ground truth.
  • Language dependency: The current heuristics and the AuToBi tool are specific to English.
  • Future Work:
    • Refine heuristics to further relax language dependency.
  • Explore the feasibility of using pre-trained acoustic prosodic labelers across languages.
  • Investigate phrasing cues that are robust across languages (e.g., silence duration).
  • Potentially use the English AuToBi labeler on other datasets to create candidate estimates of phrasing for other languages.

Summary updated

BibTeX
@misc{rendel2017weakly,
  author = {Asaf Rendel and Raul Castro Fernandez and Zvi Kons and Andrew Rosenberg and Ron Hoory and Bhuvana Ramabhadran},
  title = {{Weakly-Supervised Phrase Assignment from Text in a Speech-Synthesis System Using Noisy Labels}},
  year = {2017},
  doi = {10.21437/interspeech.2017-487},
}
Figure