TL;DR
This paper presents a rule-based approach to simulate various phonation types (like breathy, tense, creaky) using a formant synthesizer. They demonstrate that these synthesized voice qualities can be distinguished by listeners and evoke emotional impressions, suggesting a path towards more expressive synthetic speech.
What Problem Does This Paper Address?
While unit selection synthesis has significantly improved the intelligibility and naturalness of synthetic speech in limited domains, simulating diverse speaking styles and emotional nuances remains a significant challenge. Specifically, generating different voice qualities, beyond basic intelligibility, requires a deeper understanding and modeling of the underlying speech production mechanisms. Formant synthesis, which models speech production using source-filter theory, offers a parametric approach to directly manipulate these mechanisms. However, previous parametric methods often suffered from unnaturalness due to insufficient models of the voice source and its interaction with the vocal tract filter. This paper aims to bridge this gap by developing a rule-based system for generating distinct phonation types within a formant synthesis framework, building upon established phonetic descriptions of voice quality.
Methods
The authors employ a hybrid (rule-based and unit concatenating) formant synthesizer, building upon the Sensyn Klatt-style synthesizer. The core of their approach lies in manipulating the parameters of a revised Klatt glottal source model to simulate different phonation types. They leverage John Laver's phonetic descriptions of voice quality and insights from prior research to define specific parameter adjustments for each phonation type.
Key parameters modified include:
- Fo (Fundamental Frequency): Affects pitch.
- AV (Amplitude of Voicing): Relates to the overall energy of the voiced component.
- OQ (Open Quotient): The ratio of the glottal open phase to the total period duration, influencing breathiness and tenseness.
- TL (Spectral Tilt): Affects the damping of higher frequencies, contributing to breathiness and falsetto.
- AH (Amplitude of Aspiration): Represents the energy of the aspiration noise, crucial for breathy and whispery voices.
- DI (Diplophonic Double Pulsing): Used to simulate creaky voice by introducing irregular pulsing.
- FL (Fo-Flutter): Simulates jitter by randomly shifting period duration, used for falsetto.
For each phonation type (breathy, tense, whispery, creaky, falsetto), specific formulas are provided to adjust these parameters based on a "rate" parameter, indicating the degree of the phonation type. For instance, breathy voice involves increasing aspiration noise (AH) and modifying the open quotient (OQ) and spectral damping (TL). Tense voice involves increasing voiced amplitude (AV) and reducing formant bandwidths. Whispery voice is simulated by replacing the voiced component with glottal noise. Creaky voice utilizes diplophonic double pulsing (DI) and modifies the open quotient. Falsetto involves a high fundamental frequency (Fo) and strong spectral damping (TL).
Datasets
The paper does not explicitly mention training datasets. However, it refers to a previously created database of German emotional speech ([16]) and uses this as a basis for their perceptual evaluation. The audio samples generated by their synthesis system are made available for download.
Key Insights and Ideas
- Phonetic Descriptions as Synthesis Rules: The paper demonstrates the practical utility of established phonetic descriptions of voice quality (like Laver's) as a direct basis for parameter manipulation in a formant synthesizer.
- Rule-Based Control for Expressivity: By defining specific rules for parameter changes, the system can generate a range of distinct voice qualities, moving beyond simple intelligibility towards expressive speech.
- Perceptual Relevance of Synthesized Qualities: The listener experiment confirms that the synthesized phonation types are not only acoustically distinct but also capable of evoking emotional impressions, aligning with theoretical predictions.
- Sadness as a Complex Emotion: The results suggest that "sadness" might be too broad a category, with listeners distinguishing between "quite sadness" (low arousal) and "whiny sadness" (high arousal), which the synthesized voice qualities could potentially map to.
- Formant Synthesis Renaissance: The authors argue that despite the recent dominance of data-driven methods, formant synthesis, with improved models and understanding of speech mechanisms, is poised for a resurgence, particularly in hybrid systems.
Key Results
A listening experiment with 30 German native speakers was conducted. Participants were asked to assign synthesized stimuli (modal, breathy, creaky, tense, falsetto) to emotional categories (neutral, anger, joy, sadness, frightened, bored).
- Distinguishable Phonation Types: The synthesized phonation types were statistically distinguished by listeners.
- Emotional Correlations:
- Breathy and Creaky voices: Primarily associated with sadness and boredom.
- Falsetto voice: Associated with fright and, surprisingly, a high-arousal form of sadness ("whiny sadness").
- Tense voice: Strongly promoted an angry speaker impression, but also elicited a high rating for sadness.
- Modal voice: Did not elicit a statistically significant emotional impression, except that it clearly did not sound angry, which is the desired outcome for a neutral baseline.
Limitations and Future Work
- "Tense Voice" as a Global Setting: The authors acknowledge that "tense voice" is not purely a phonation type but also involves supralaryngeal components. They use it as a global setting, aligning with other authors, and note its correlation with the arousal dimension of emotion.
- Creaky Voice Variability: They mention that creaky voice can be variable and sometimes associated with lower arousal, leading them to distinguish "lax creaky voice" which they model with specific parameter adjustments.
- Falsetto Noise Components: The authors did not model the added noise components in falsetto voice, attributing this to their weakness at low subglottal pressure.
- "Sadness" Granularity: The study suggests that the emotional category of "sadness" might need to be refined into sub-categories based on arousal levels.
- Future Work: The paper concludes by reiterating the potential of formant synthesis in hybrid systems (pseudo-articulatory or formant-driven unit concatenation) for creating more expressive synthetic speech. The synthesis system and its source code are available for download, encouraging further research.
Summary updated