TL;DR
This paper introduces a novel HMM-based prosodic structure model that leverages deep syntactic features extracted from French text. By incorporating higher-level linguistic information like syntactic chunks and adjunctions, the model significantly improves the accuracy of predicting prosodic boundaries and prominence, demonstrating the crucial link between complex syntax and speech intonation.
What Problem Does This Paper Address?
Synthesized speech, while increasingly natural-sounding, often lacks the expressive nuance of human speech, primarily due to limitations in modeling prosody – the rhythm, stress, and intonation of speech. Accurately predicting prosodic structure is crucial for generating more natural and engaging synthesized speech, especially for applications like multimedia and artistic productions. Previous attempts to model prosody using linguistic information often relied on simpler, surface-level features. This paper addresses the gap by exploring how deep syntactic analysis, which captures more complex grammatical relationships within a sentence, can be used to build more robust and accurate prosodic structure models.
Methods
The core of this work is a Hidden Markov Model (HMM) based prosodic structure model. The novelty lies in the rich linguistic context used to train and predict this model. The authors developed a sophisticated linguistic processing chain for French that goes beyond basic part-of-speech tagging. This chain performs:
- Morpho-syntactical labeling: Extracting basic word-level features.
- Deep syntactic parsing: Using the FRMG parser to build a detailed dependency graph that represents grammatical relationships between words and phrases.
From this deep parse, they extract several sets of syntactic features, ranging from simple form dependencies to more complex representations of:
- Recursive chunks: Identifying nested grammatical units.
- Adjunctions: Capturing phenomena like parentheticals, subordinate clauses, and enumerations, which often have distinct prosodic implications.
These feature sets are then used to train HMMs, where the model learns to predict prosodic labels (like major and minor frontiers, and lexical prominence) based on the sequence of linguistic features. The prediction is done using a Viterbi algorithm for maximum likelihood estimation.
Datasets
The study uses two distinct French speech corpora:
- Laboratory corpus: A smaller, controlled corpus recorded in an anechoic room by naive speakers. This corpus has simpler linguistic and prosodic complexity.
- Multi-media corpus: A larger corpus featuring a professional actor reading literary text. This corpus exhibits higher linguistic and prosodic complexity, with more varied and less stereotypical prosodic strategies.
The paper notes that automatic linguistic feature extraction was less robust for the multi-media corpus (achieving complete parsing for 52% of sentences compared to 80% for the laboratory corpus), which is acknowledged as a limitation.
Key Insights and Ideas
- Deep syntax matters for prosody: The most significant finding is that higher-level syntactic features, particularly those related to chunks and adjunctions, dramatically improve prosodic structure prediction. This highlights a strong, previously under-exploited, link between complex grammatical structures and how we intone our speech.
- Surface features are insufficient: Simple morpho-syntactical features and form dependencies alone do not yield substantial improvements, suggesting that basic word-level information is not enough to capture prosodic nuances.
- Prosodic frontiers vs. prominence: The model performs much better at predicting prosodic frontiers (boundaries between prosodic groups) than lexical prominence (stress on individual words). This is attributed to frontiers being more directly tied to syntactic structure, while prominence is influenced by semantic and discursive factors that are harder to capture with syntax alone.
- Actor speech is challenging: The performance drop observed with the professional actor's corpus underscores the difficulty of modeling highly complex and varied prosodic strategies, as well as the limitations of current deep parsing techniques on such data.
Key Results
The paper quantifies the improvements using two metrics: Bayesian Information Criterion (BIC) and Weighted Cohen's Kappa.
- Significant improvements with higher-level features: For the laboratory corpus, using the adjunction feature set led to a 21% reduction in Kappa error and a 19% reduction in BIC. Similar improvements were seen for the multi-media corpus (11% Kappa reduction, 7.5% BIC reduction).
- Chunk and adjunction features are key: The performance gains were most pronounced when incorporating chunk and adjunction features, confirming their importance.
- Performance drop on complex data: The multi-media corpus consistently showed lower performance across all feature sets compared to the laboratory corpus, indicating the challenges of real-world, complex speech.
Limitations and Future Work
- Lexical prominence prediction: The model struggles to accurately predict lexical prominence, which the authors attribute to its strong ties to semantic and discursive levels, suggesting that future work should incorporate these levels.
- Hierarchical modeling: The current HMM framework is sequential. The authors propose exploring hierarchical HMMs or Weighted Tree Automata (WTA) to better model the hierarchical nature of prosodic structure.
- Robustness on complex data: The performance drop on the multi-media corpus highlights the need for more robust linguistic parsers and prosodic modeling techniques for complex, naturalistic speech.
- Semantic and discursive features: To improve prominence prediction, future research will focus on introducing semantic and discursive information into the model.
Summary updated