A discriminative approach to phrase break modelling

Stephen CoxInterspeech 2005

TL;DR

This paper introduces a discriminative approach to predicting pauses between words in speech, crucial for natural-sounding text-to-speech systems. By applying the Generalized Probabilistic Descent (GPD) algorithm to both Naive Bayes and Maximum Entropy models, the authors significantly improve pause prediction accuracy, achieving state-of-the-art results on the Boston Radio Corpus.

What Problem Does This Paper Address?

Generating natural-sounding speech from text (Text-to-Speech, TTS) requires more than just pronouncing words correctly; it also involves placing appropriate pauses. Predicting these pauses, or "phrase breaks," is a challenging task. While some pauses are dictated by punctuation, many are not, and their placement often depends on semantic and pragmatic understanding of the sentence. This paper focuses on a pattern classification approach to predict whether a juncture between two words should be a pause or not, aiming to improve the naturalness of synthesized speech.

Methods

The authors compare two classification frameworks: Naive Bayes (NB), a generative model, and Maximum Entropy (ME), a discriminative model. They then introduce the Generalized Probabilistic Descent (GPD) algorithm, a gradient-based method designed to iteratively adjust model parameters to minimize classification error on training data. The core contribution is applying GPD to adapt both NB and ME models.

  • Feature Engineering: They use a combination of local features (derived from words surrounding a juncture) and global features (derived from a sentence's parse tree). They also employ a reduced set of Part-of-Speech (PoS) tags to improve feature utility.
  • Naive Bayes (NB): A generative model that assumes independence between features.
  • Maximum Entropy (ME): A discriminative model that directly models class posteriors and is known to perform well when feature independence assumptions are violated.
  • Generalized Probabilistic Descent (GPD): An iterative algorithm that adjusts model weights to directly minimize classification error. The paper highlights that GPD is particularly well-suited for the ME framework, acting as a "harder" discriminative approach compared to ME's "softer" approach.

Datasets

  • Boston Radio News Corpus: This corpus, annotated with ToBI prosody labels, was used for training and testing.
  • Training set: 13,754 words (3,437 breaks)
    • Testing set: 15,333 words (3,894 breaks)
    • Breaks were defined as junctures labeled level 3 or above in the ToBI specification.

Key Insights and Ideas

  • GPD Significantly Boosts Performance: Applying GPD to both NB and ME models led to substantial performance improvements, demonstrating the algorithm's effectiveness in refining existing models.
  • ME Outperforms NB (Initially): Untransformed ME weights consistently outperformed NB probabilities, especially as the number of attributes increased. This suggests that the NB assumption of feature independence is often violated in this task.
  • GPD is a Strong Starting Point for ME: The authors hypothesize that ME's "softer" discriminative approach provides a better starting point for GPD than a generative model like NB. Their results support this, showing significant gains when GPD is applied to ME weights.
  • Punctuation is a Weak Baseline: Using punctuation alone as a break predictor yielded a low F-score, highlighting the need for more sophisticated methods.
  • Error Analysis Reveals Deeper Challenges: Misclassifications (both breaks missed and non-breaks identified as breaks) often point to the need for semantic understanding and long-range syntactic context, which are beyond the scope of the current feature set.

Key Results

  • State-of-the-Art Performance: The best result achieved was an F-score of 81.2% with a juncture error rate of 10.24% after applying GPD to the ME model using a specific set of attributes (2, 3, 6, and 7). This is reported as the best performance on the Boston Radio Corpus at the time of publication.
  • GPD Improvement: For the ME model, GPD adaptation led to an average F-score increase from 68.24% to 78.24% across different attribute sets. For NB, the improvement was from 26.69% to 64.80%.
  • Baseline Performance: Using punctuation for break prediction resulted in an F-score of 51.72%.

Limitations and Future Work

  • Limitations:
    • Some misclassifications (especially breaks missed) indicate a need for deeper semantic and long-range syntactic understanding, which the current feature set does not fully capture.
  • The NB model's performance degrades with more attributes, confirming the violation of its independence assumption.
  • Future Work:
    • Further investigation into the relationship between ME and GPD discrimination.
  • Applying GPD to other language processing tasks.
  • Exploring more sophisticated features that capture semantic and long-range syntactic dependencies to address the limitations observed in error analysis.

Summary updated

BibTeX
@inproceedings{cox2005discriminative,
  author = {Stephen Cox},
  title = {{A discriminative approach to phrase break modelling}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure