TL;DR
This paper introduces the Affective Story Teller (AST), a Text-to-Speech (TTS) system designed to imbue synthesized speech with emotional expressivity. AST tackles the limitations of current TTS systems in conveying emotion by integrating linguistic analysis of input text with prosodic parameterization and pitch manipulation, leading to significantly improved emotional perception by listeners.
What Problem Does This Paper Address?
Synthesized speech, while increasingly natural, often lacks the emotional nuance that makes human communication engaging and effective. Existing Text-to-Speech (TTS) systems struggle to capture and convey emotions, resulting in monotonous or inappropriate vocalizations. This paper addresses the gap in creating TTS systems that can generate emotionally expressive speech by focusing on how to better parameterize acoustic and prosodic features based on the emotional content of the input text. The authors highlight that while some TTS systems can accept markup for prosody, few intelligently process the text to derive these emotional cues.
Methods
The AST system employs a multi-stage approach:
- Affective Information Processing: The system first analyzes the input text to identify its emotional content. This involves:
- Syntactic and Functional Dependency Parsing: Using the Machinese Syntax parser to break down sentences into triplets (subject, verb, object) with associated attributes.
- SenseNet Analysis: Leveraging a system called SenseNet (developed by Shaikh et al.) which processes these triplets to perform sentence-level affective sensing. SenseNet assigns a "sentence-valence" score and can classify texts into eight emotion types (happy, sad, hope, fear, admiration, shame, love, hate) based on the OCC emotion model. The authors map these to five basic emotions: happy, sad, fear, anger, and surprise.
- MaryXML Generation: Based on the identified emotion and its valence, the system dynamically generates MaryXML scripts. These scripts contain specific prosodic parameters (e.g., speech rate, pitch average, pitch range, intensity, pitch dynamics) tailored to the detected emotion, drawing on established research on human vocal emotion.
- TD-PSOLA based Modifications: The generated MaryXML is fed into the MARY TTS system for initial speech synthesis. Subsequently, the synthesized speech samples undergo further modification using the Time Domain Pitch Synchronous Overlap and Add (TD-PSOLA) algorithm. This stage fine-tunes prosodic parameters like F0 mean, range, and shape to more accurately reflect the desired emotional expression. For instance, "happy" speech might have a higher F0 mean and wider pitch range, while "sad" speech would have a lower F0 mean and narrower range.
Datasets
The paper does not explicitly mention specific datasets used for training the SenseNet system. However, for evaluation, they used a folk tale containing sentences categorized by emotion: 12 happy, 12 sad, 12 anger, 12 fear, 6 surprise, and 9 neutral emotion-bearing sentences. The synthesized speech samples were evaluated by 12 non-English speaking native participants.
Key Insights and Ideas
- Bridging Textual Emotion and Vocal Prosody: The core innovation lies in the intelligent pipeline that translates textual emotional cues into concrete prosodic parameters for TTS. This moves beyond simple markup and into a deeper understanding of how text content dictates vocal expression.
- SenseNet's Role in Affective Sensing: The use of SenseNet for sentence-level affective analysis, based on semantic triplets and a cognitive emotion model, provides a robust mechanism for extracting emotional information from raw text.
- Dynamic MaryXML Generation: The ability to dynamically create MaryXML scripts tailored to specific emotions is crucial. This allows for fine-grained control over synthesis parameters, which is essential for nuanced emotional expression.
- TD-PSOLA for Refinement: Employing TD-PSOLA for post-synthesis modification allows for precise adjustments to pitch and its dynamics, further enhancing the emotional quality of the synthesized speech.
- Human Perception is Key: The paper emphasizes that the ultimate measure of success is human perception. Their experimental results validate that their approach significantly improves the listener's ability to discern emotions in synthesized speech.
Key Results
A perceptual test was conducted where participants listened to synthesized speech from a standard TTS system (S1) and the AST system (S2), as well as human-spoken speech (R).
- Improved Emotion Perception: The results showed that listeners could perceive "happy," "sad," and "fear" emotions significantly better from AST-synthesized speech (S2) compared to standard TTS (S1).
- Confusion Matrix Analysis: Table 2 (Confusion Matrix) and Table 3 (Precision, Recall, F-Measure) quantitatively demonstrate this improvement. For instance, the average accuracy for S2 was 58.73%, significantly higher than S1's 47.49%, and closer to the human-spoken baseline (R) which achieved an average accuracy of 70.50%.
- Addressing Weaknesses: The study found that S1 was particularly weak at conveying positive emotions like "happy" and "surprise" and tended to add negative connotations. AST (S2) successfully addressed these issues.
Limitations and Future Work
- Limitations:
- The paper acknowledges that the current system is still "weak to convey positive emotions" and has a "tendency to add negative emotion" in some cases, though S2 shows improvement over S1.
- The evaluation was conducted with non-English speaking natives, which might influence perception nuances.
- The system currently relies on pre-defined mappings and rules for emotion to prosody.
- Future Work:
- The authors plan to build a comprehensive tool that integrates all discussed resources and offer it as an add-on to the MARY TTS system.
- They aim to incorporate the ability to handle different levels of negativism/positivism within different phrases of a sentence for even better emotion perceivedness.
- Further research into more sophisticated emotion recognition from text and more dynamic prosodic control is implied.
Summary updated