TL;DR
This paper introduces VOKinesis, a novel system for real-time control of speech rhythm and intonation using gestural input. By mapping hand gestures to "arsis" (transient speech segments) and "thesis" (vocalic nuclei), the system allows for intuitive manipulation of speech timing, opening new avenues for expressive speech synthesis and prosodic research.
What Problem Does This Paper Address?
Traditionally, speech synthesis has focused on generating intelligible speech. However, capturing and controlling the nuanced rhythm and intonation (prosody) that makes speech expressive and natural remains a significant challenge. While previous work has explored gestural control for intonation, controlling speech rhythm in real-time with similar intuitive methods has been less developed. This paper addresses this gap by proposing a system that allows users to "play" speech rhythm like a musical instrument, using gestural input to manipulate the timing of speech segments. This is crucial for creating more lifelike synthetic speech and for gaining deeper insights into how humans produce and perceive speech rhythm.
Methods
The VOKinesis system operates on pre-recorded and labeled speech signals, modifying them in real-time using the RT-PSOLA (Real-Time Pitch Synchronous Overlap-Add) technique. The core innovation lies in the biphasic performative control of speech rhythm. This control is achieved through two main gestural inputs:
- Pitch Control: The preferred hand uses a stylus on a graphic tablet to control pitch, similar to drawing. This leverages existing research showing the effectiveness of such methods for intonation control.
- Rhythm Control: The non-preferred hand controls rhythm using a button. This control is based on the "frame/content theory" of speech production, which posits that speech is organized into syllabic frames (rhythmic cycles) and segmental content (phonemes). The system defines two key points within these frames:
- Arsis: Represents transient segments, typically the coda of one syllable and the attack of the next, or the transition between vowels.
- Thesis: Represents the strong beat of the syllable, usually the vocalic nucleus.
These "Syllabic Control Points" (SCP) are mapped to temporal targets. The system uses two modes for rhythm control:
- Tap Mode: A button press triggers a vocalic phase (thesis), and release triggers a transient phase (arsis). This creates a binary control over rhythm.
- Fader Mode: (Briefly mentioned, not the focus of the experiment) Allows for continuous control of transient phase timing, deemed more suitable for singing synthesis.
Datasets
The paper does not describe the use of pre-existing datasets for training. Instead, it uses a set of 8 sentences (ranging from 2 to 9 syllables) recorded by a male and a female speaker. These recordings serve as the basis for the VOKinesis system and for the experimental evaluation.
Key Insights and Ideas
- Intuitive Biphasic Rhythm Control: The mapping of "arsis" and "thesis" to distinct gestural inputs (button press/release) provides a surprisingly intuitive way to control speech rhythm.
- Frame/Content Theory in Practice: The system successfully operationalizes the frame/content theory by using "arsis" and "thesis" as control points for syllabic frames, demonstrating its practical utility.
- Rhythm Control is Largely Independent of Pitch Control: The experimental results suggest that accurate rhythm reproduction can be achieved even without explicit pitch control, highlighting the distinct nature of these prosodic parameters.
- Potential for Speech Therapy: The gestural and rhythmic control mechanisms show promise for developing new speech therapy techniques, particularly for conditions like Broca's aphasia or stuttering, by leveraging audio-motor synchronization.
Key Results
The VOKinesis system was evaluated in a prosodic imitation experiment where subjects attempted to reproduce sentence prosody using the system.
- Rhythm Accuracy: On average, inter-vocalic durations (a key measure of rhythm) were reproduced with an average difference of approximately 20 ms. This is considered excellent accuracy, corresponding to about two periods at a 100 Hz pitch.
- Pitch Control Impact: Pitch control did not significantly improve rhythm reproduction accuracy.
- Speaker Impact: Neither the male nor the female original speaker's prosody significantly affected the subjects' rhythm reproduction accuracy.
- Syllable Length: Longer syllables (> 220 ms) were found to be slightly more difficult to reproduce accurately, with mean differences reaching up to 50 ms. Subjects tended to shorten long syllables and lengthen short ones.
- Variability Preservation: Despite the tendency to adjust durations, subjects were able to follow the original inter-vocalic duration variability within a sentence.
Limitations and Future Work
- Tap Mode Limitations: The paper acknowledges that the "Tap mode" for rhythm control, while effective, might not be as precise as continuous control methods like faders, especially for very fine articulation timing.
- Future Work on Continuous Control: The authors plan to explore rhythm control using continuous controllers, such as foot pedals, for both speech and singing synthesis. They also aim to investigate which control methods are best suited for different applications.
- Musical Applications: The system's potential for musical improvisation and performative singing synthesis is highlighted, with examples provided.
- Brain Imaging Studies: The authors suggest that VOKinesis could be a valuable tool for brain imaging studies to investigate theories of voice production and motor planning.
- Speech Therapy Applications: Further research is proposed to explore the system's utility in speech therapy, particularly for patients with aphasia or stuttering.
Summary updated