TL;DR
This paper introduces a novel method for generating speech prosody (pitch and duration) directly from syntax trees, bypassing traditional rule-based systems. By treating syntax trees as a direct source of prosodic information and using a statistical learning approach (CART), the system learns to map syntactic structures to prosodic contours, leading to more natural-sounding speech synthesis.
What Problem Does This Paper Address?
Generating natural-sounding prosody in Text-to-Speech (TTS) systems is a long-standing challenge. While unit selection methods can leverage existing prosody from speech corpora, systems that build speech from smaller units (like diphones or HMMs) need to explicitly generate prosody for each segment. Traditional approaches often rely on complex, language-specific rule systems derived from linguistic theories. These systems, while effective, are difficult to create, maintain, and adapt to new languages. Statistical methods, like HMMs, offer language independence but often struggle to capture long-range dependencies in speech prosody due to limited context. This paper aims to bridge this gap by using the rich structural information present in syntax trees to drive a statistical prosody generation process, offering a more robust and potentially more natural solution.
Methods
The core innovation lies in treating the syntax tree itself as the primary input for prosody generation. Instead of extracting specific features or applying transformation rules, the authors propose a "superpositional model" where each node in the syntax tree acts as a "mutator function." These mutator functions learn to contribute to the prosody contour of syllables within their subtree.
The process involves:
- Syntax Tree Simplification: The original syntax trees are simplified by removing most attributes, retaining only constituent types. Words become leaves, and syllabification is performed to make syllable information available.
- Mutator Functions: Each node in the simplified syntax tree is assigned a mutator function. This function takes the current prosody contour of a syllable and modifies it based on the node's constituent type and its context within the tree.
- Recursive Superposition: The prosody contour of a syllable is computed by recursively applying the mutator functions of all nodes on the path from the root of the tree to the leaf node corresponding to that syllable. This allows for a hierarchical influence of syntax on prosody.
- Contextual Features: To make the mutator functions effective, they are trained using a rich set of contextual features, including parent constituent, sibling constituents, punctuation, number of syllables, and lexical stress.
- CART Regression: Classification and Regression Trees (CART) are used as the learning algorithm for the mutator functions. This choice is motivated by CART's ability to handle correlated and irrelevant features.
- Analysis-Synthesis Learning: Due to the dependency of mutator functions on intermediate outputs, an iterative analysis-synthesis approach is used for training. Mutators are applied, intermediate contours are computed, and then new mutators are trained based on these contours.
Datasets
The paper mentions using a German speech corpus for training and evaluation. This corpus consists of 1573 sentences spoken by a single professional female speaker, including statements with phenomena like address, numbers, and enumerations. The sentences were spoken in isolation with a lively speaking style. No new benchmarks were introduced; the evaluation was against existing rule-based and baseline ANN models.
Key Insights and Ideas
- Syntax as a Direct Prosody Generator: The most significant idea is that the hierarchical structure of a syntax tree can directly inform prosody generation without needing an intermediate abstract prosodic representation. This leverages the inherent structure of language to guide speech naturalness.
- Mutator Functions for Hierarchical Influence: The concept of "mutator functions" at each syntax node is elegant. It allows for a granular, yet hierarchical, contribution of syntactic elements to the final prosody, mimicking how different levels of linguistic structure influence speech.
- Statistical Learning for Language Independence: By using CART to learn the mutator functions, the system gains language independence. Once trained on a corpus, the same framework can theoretically be applied to different languages, provided syntax trees are available.
- Sensitivity to Syntax Analysis Errors: A crucial insight is that this method makes the system more sensitive to errors in the syntax analysis stage. While rule-based systems might ignore such errors, the statistical approach can highlight them through unnatural prosody, acting as a form of error detection.
Key Results
The paper reports that the CART-based prosody generation system was able to learn duration equally well as the rule-based system, achieving a training error of 20.7 ms. For F0, it performed slightly worse with an error of 30.5 Hz compared to the rule-based system's 29.7 Hz, which the authors attribute to the ANN models' recurrent structure.
Qualitatively, manual inspection showed that the CART-based model largely agreed with the rule-based system on primary phrase boundaries but differed on secondary ones. Notably, the CART system learned to place the main accent on the first noun phrase and introduce minor phrase boundaries, demonstrating its ability to capture more nuanced prosodic structures than the rule-based system in this specific example.
Limitations and Future Work
- Training Data Dependency: The authors acknowledge that the largest limitation was the limited training material. While sufficient for training ANN models in the rule-based system, it was less so for the mutator functions. A larger and more varied corpus is suggested to enable the CART-based system to generate richer prosody.
- Sensitivity to Syntax Errors: While presented as a potential benefit for error detection, the increased sensitivity to syntax analysis errors means that the quality of the prosody is directly tied to the accuracy of the syntax parser.
- Future Work: The authors suggest that phenomena like negation and contrast, which have a strong influence on prosody, could be easily formulated within the context of a DCG grammar framework. This implies future work could focus on integrating more linguistically motivated syntactic structures to generate specific prosodic effects. They also mention that the grammar used in the prototype was not specially tuned for prosodically relevant structures, hinting at potential improvements through grammar engineering.
Summary updated